United  States  Government  Accountability  Office 

GKO 

Report  to  the  Committee  on  Armed 
Services,  House  of  Representatives 

September  2014 

F-35  SUSTAINMENT 

Need  for  Affordable 
Strategy,  Greater 
Attention  to  Risks, 
and  Improved  Cost 
Estimates 

GAO-14-778 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

SEP  2014  2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2014  to  00-00-2014 

4.  TITLE  AND  SUBTITLE 

F-35  Sustainment:  Need  for  Affordable  Strategy,  Greater  Attention  to 
Risks,  and  Improved  Cost  Estimates 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Government  Accountability  Office, 441  G  Street 

NW, Washington, DC, 20548 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  aS 

unclassified  unclassified  unclassified  Report  (SAR) 

60 

Standard  Form  298  (Rev.  8-98} 

Prescribed  by  ANSI  Std  Z39-18 


September  2014 


GM) 

Highlights 

Highlights  of  GAO-14-778,  a  report  to  the 
Committee  on  Armed  Services,  House  of 
Representatives 


F-35  SUSTAINMENT 

Need  for  Affordable  Strategy,  Greater  Attention  to 
Risks,  and  Improved  Cost  Estimates 


Why  GAO  Did  This  Study 

The  F-35  Lightning  il  is  intended  to 
replace  a  variety  of  existing  aircraft  in 
the  Air  Force,  Navy,  and  Marine  Corps, 
while  providing  the  most  supportable, 
technologically  advanced,  lethal,  and 
survivable  aircraft  to  date.  The  F-35  is 
DOD’s  most  expensive  weapon 
system,  with  estimated  sustainment 
costs  of  about  $1  trillion.  With  the 
military  services  planning  for  the  ability 
to  deploy  and  maintain  the  F-35  within 
4  years,  DOD  is  working  to  develop  a 
sustainment  strategy  that  will  be  both 
affordable  and  executable  for  the 
program’s  life  cycle. 

GAO  was  mandated  to  review  DOD’s 
F-35  sustainment  planning  efforts.  This 
report  addresses  the  extent  to  which 
DOD  has  (1)  developed  an  F-35 
sustainment  strategy  and  addressed 
potential  risks  related  to  affordability 
and  operational  readiness  and  (2) 
developed  a  reliable  O&S  cost 
estimate  for  the  program’s  life  cycle. 
GAO  analyzed  documented  plans  and 
cost  estimates  and  interviewed  DOD 
and  contractor  officials. 

What  GAO  Recommends 

GAO  recommends  that  DOD  develop 
better  informed  affordability 
constraints;  address  three  risks  that 
could  affect  sustainment,  affordability, 
and  operational  readiness;  and  take 
steps  to  improve  the  reliability  of  its 
cost  estimates.  DOD  concurred  with  all 
but  one  recommendation  and  partially 
concurred  with  the  recommendation  to 
conduct  uncertainty  analysis  on  one  of 
its  cost  estimates,  stating  it  already 
conducts  a  form  of  uncertainty 
analysis.  GAO  continues  to  believe 
that  the  recommended  analysis  would 
provide  a  more  comprehensive  sense 
of  the  uncertainty  in  the  estimates. 

View  GAO-14-778.  For  more  information, 
contact  Cary  Russell  at  (202)  512-5431  or 
russellc@gao.gov. 


What  GAO  Found 

The  Department  of  Defense  (DOD)  currently  has  or  is  developing  several  plans 
and  analyses  that  will  make  up  its  overall  F-35  sustainment  strategy,  which  is 
expected  to  be  complete  in  fiscal  year  2019.  The  annual  F-35  operating  and 
support  (O&S)  costs  are  estimated  to  be  considerably  higher  than  the  combined 
annual  costs  of  several  legacy  aircraft  (see  fig.).  DOD  has  begun  some  cost- 
savings  efforts  and  established  sustainment  affordability  targets  for  the  F-35 
program,  but  DOD  did  not  use  the  military  services’  budgets  to  set  these  targets. 
Therefore,  these  targets  may  not  be  representative  of  what  the  services  can 
afford  and  do  not  provide  a  clear  benchmark  for  DOD’s  cost-savings  efforts.  In 
addition,  DOD  has  not  fully  addressed  several  issues  that  have  an  effect  on 
affordability  and  operational  readiness,  including  aircraft  reliability  and  technical- 
data  rights,  which  could  affect  the  development  of  the  sustainment  strategy. 


Comparison  of  the  Annual  Estimated  F-35  Operating  and  Support  (O&S)  Cost  at  Steady  State 
to  Actual  Legacy  Aircraft  O&S  Costs  in  Fiscal  Year  2010 
Aircraft 


0  5  10  15  20 

Billions  of  dollars 

Source:  GAO  presentation  of  Department  of  Defense  data  and  Air  Force  and  Marine  Corps  photos.  I  GAO-14-778 

Notes:  For  the  purposes  of  this  report,  GAO  defines  steady-state  operations  as  the  period  from  2036 
to  2040,  when,  according  to  the  services’  plans,  the  number  of  F-35  aircraft  and  flying  hours  reaches 
its  highest  point  and  plateaus. 

’’The  F-35  cost  presented  is  Cost  Assessment  and  Program  Evaluation’s  (CAPE)  estimated  total 
annual  operating  and  support  (O&S)  cost  for  2040  in  base  year  2012  dollars. 

‘’Legacy  aircraft  cost  is  based  on  a  CAPE  analysis  of  2010  cost  data,  representing  a  high  point  for 
aircraft  O&S  budgets  due  to  contingency  operations  at  that  time. 

It  is  unclear  whether  DOD’s  O&S  cost  estimates  for  the  F-35  program  reflect  the 
most  likely  costs  that  the  F-35  program  will  incur.  DOD  has  two  primary  F-35 
O&S  estimates  that  each  total  around  $1  trillion  over  a  56-year  life  cycle.  These 
cost  estimates  are  comprehensive  in  that  they  include  all  DOD-required  program 
elements  and  are  organized  according  to  a  standard  O&S  cost-estimating 
structure;  however,  weaknesses  exist  with  respect  to  a  few  of  the  assumptions, 
and  the  estimates  did  not  include  all  analyses  necessary  to  make  them  fully 
reliable.  For  example,  the  estimates  did  not  use  reasonable  fuel  burn  rate 
assumptions  that  reflect  the  likely  future  F-35  fuel  usage.  Further,  one  of  the 
estimates  did  not  use  reasonable  assumptions  about  part  replacement  rates  and 
depot  maintenance.  Finally,  while  DOD  took  some  steps  to  mitigate  the 
uncertainties  inherent  in  cost  estimates,  DOD  officials  did  not  conduct  key 
analyses  to  determine  the  level  of  risk  associated  with  the  estimates. 
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Abbreviations 

APRS 

Anomaly  and  Pailure  Resolution  System 

ALIS 

Autonomic  Logistics  Information  System 

AT&L 

Acquisitions,  Technology,  and  Logistics 

CAPE 

Cost  Assessment  and  Program  Evaluation 

CTOL 

conventional  takeoff  and  landing 

CV 

carrier-suitable  variant 

□PARS 

Defense  Pederal  Acquisition  Regulation  Supplement 

DOD 

Department  of  Defense 

DOT&E 

Director  of  Operational  Test  and  Evaluation 

EOTS 

Electrical  Optical  Targeting  System 

PAR 

Pederal  Acquisition  Regulation 

IOC 

initial  operational  capability 

IP 

Intellectual  Property 

JPO 

Joint  Program  Office 

JRMET 

Joint  Reliability  and  Maintainability  Evaluation  Team 

KPP 

Key  Performance  Parameter 

MPHBR 

Mean  Plight  Hours  Between  Removals 

O&S 

operating  and  support 

OSD 

Office  of  the  Secretary  of  Defense 

OSD-LMR 

Office  of  the  Secretary  of  Defense  Logistics  and 

Materiel  Readiness 

PBL 

Performance  Based  Logistics 

R+M 

reliability  and  maintainability 

SAR 

Selected  Acquisition  Report 

SOS 

System  of  Systems 

STOVL 

short  takeoff  and  vertical  landing 
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U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


September  23,  2014 

The  Honorable  Howard  P.  “Buck”  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Department  of  Defense  (DOD)  has  a  continuing  responsibility  to 
procure  weapon  systems  that  successfully  execute  its  national  security 
mission.  One  such  weapon  system  is  the  F-35  Lightning  II — also  known 
as  the  Joint  Strike  Fighter — which  is  intended  to  replace  a  variety  of 
existing  aircraft  in  the  Air  Force,  Navy,  and  Marine  Corps,  while  providing 
the  most  technologically  advanced,  lethal,  supportable,  and  survivable 
aircraft  to  date.  With  DOD  estimating  its  sustainment  costs  to  be 
approximately  $1  trillion  over  its  life  cycle,  the  F-35  is  not  only  the  most 
ambitious  weapon  system  in  DOD’s  history,  but  also  the  most  costly. 
Recognizing  that  at  least  70  percent  of  a  weapon  system’s  life-cycle  costs 
stem  from  operating  and  supporting  the  system,  in  recent  years  DOD  has 
made  changes  to  its  acquisition  process  to  put  an  earlier  emphasis  on 
sustainment.  Specifically,  more  attention  has  been  placed  on  sustainment 
planning,  assuring  competition  among  suppliers,  and  identifying 
sustainment-related  resource  constraints  at  the  front  end  of  the 
acquisition  process.  However,  the  F-35  acquisition  program,  which  began 
in  October  2001 ,  predates  these  changes,  and  DOD  has  only  recently 
begun  to  focus  on  how  it  will  sustain  the  nearly  2,500  aircraft  it  plans  to 
procure.  Currently,  the  F-35  is  13  years  into  its  acquisition  strategy — a 
strategy  that  involves  substantial  overlap  among  development,  testing, 
and  production  activities.  With  all  three  military  services  planning  to 
deploy  and  maintain  the  F-35  within  the  next  4  years,  and  the  Marine 
Corps  planning  to  do  so  in  less  than  1  year,  DOD  is  working  to  develop  a 
sustainment  strategy  for  the  F-35  that  will  be  both  affordable  and 
executable  for  the  life  cycle  of  the  program. 

We  have  reported  on  DOD’s  acquisition  of  the  F-35  for  many  years  (see 
the  Related  GAO  Products  section  at  the  end  of  this  report).  Our  body  of 
work  has  identified  significant  cost,  schedule,  and  performance  problems 
and  found  that  those  problems,  in  large  part,  can  be  traced  to  (1) 
decisions  made  at  key  junctures  without  adequate  product  knowledge; 
and  (2)  a  highly  concurrent  acquisition  strategy  with  significant  overlap 
among  development,  testing,  and  manufacturing  activities.  In  March 
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2012,  DOD  completed  an  extensive  restructuring  of  the  F-35  program  by 
increasing  the  program’s  cost  estimates,  extending  its  testing  and  delivery 
schedules,  and  deferring  near-term  aircraft  procurement  quantities  into 
the  future.  We  concluded  in  June  2012  and  in  March  2013  that  the 
restructuring  actions  should  lead  to  more  achievable  and  predictable 
outcomes,  albeit  at  higher  costs  and  with  longer  time  frames  than 
originally  planned  for  testing  and  delivering  capabilities  to  the  warfighter.'' 
In  March  2014,  we  found  that  problems  encountered  by  DOD  in 
completing  software  testing  may  hinder  delivery  of  expected  warfighting 
capabilities  to  most  of  the  services.^  We  have  made  numerous 
recommendations  aimed  at  addressing  these  issues,  and  DOD  has  taken 
some  actions  to  address  them  to  varying  degrees. 

In  light  of  DOD  cost  estimates  for  sustainment  of  the  aircraft  amounting  to 
approximately  $1  trillion,  and  increased  concerns  over  affordability  in  a 
fiscally  constrained  environment,  the  House  Armed  Services  Committee 
report  accompanying  a  National  Defense  Authorization  bill  for  Fiscal  Year 
2014  mandated  GAO  to  review  DOD’s  sustainment  planning  efforts  for 
the  F-35  program.^  This  report  addresses  the  extent  to  which  DOD  has 

(1 )  developed  a  sustainment  strategy  for  the  F-35  program  and 
addressed  potential  risks  to  affordability  and  operational  readiness  and 

(2)  developed  a  reliable  operating  and  support  (O&S)  cost  estimate  for 
the  life  cycle  of  the  program. 

For  each  of  our  objectives,  we  reviewed  relevant  policy  and  procedures 
and  collected  information  by  interviewing  officials  from  the  Office  of  the 
Under  Secretary  of  Defense  (Acquisitions,  Technology  and  Logistics),  the 
Office  of  the  Assistant  Secretary  of  Defense  (Logistics  and  Materiel 
Readiness),  the  Office  of  the  Director  for  Cost  Assessment  and  Program 
Evaluation  (CAPE),  the  Office  of  the  Director  for  Operational  Test  and 
Evaluation  (DOT&E),  the  Office  of  the  Director  for  Developmental  Test 


^GAO,  Joint  Strike  Fighter:  DOD  Actions  Needed  to  Further  Enhance  Restructuring  and 
Address  Affordabiiity  Risks,  GAO-12-437  (Washington,  D.C.:  June  14,  2012)  and  F-35 
Joint  Strike  Fighter:  Current  Outiook  Is  Improved,  but  Long-Term  Affordability  Is  a  Major 
Concern,  GAO-1 3-309  (Washington,  D.C.:  Mar.  11, 2013). 

^GAO,  F-35  Joint  Strike  Fighter:  Problems  Completing  Software  Testing  May  Hinder 
Delivery  of  Expected  Warfighting  Capabilities,  GAO-14-322  (Washington,  D.C.:  Mar.  24, 
2014). 

^H.R.  Rep.  No.  113-102,  at  118  (2013). 
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and  Evaluation,  the  Air  Force,  the  Navy,  the  Marine  Corps,  and  the  F-35 
Joint  Program  Office  (JPO).  We  also  gathered  F-35  reliability  and 
maintainability  (R+M)  data'^  from  2013  through  2014  that  had  been 
verified  through  DOD’s  Joint  Reliability  and  Maintainability  Evaluation 
Team  (JRMET).  To  determine  the  reliability  of  these  data,  we  collected 
information  on  how  the  data  were  collected,  managed,  and  used  through 
a  survey  and  interviews  and  with  relevant  DOD  officials.  In  addition  to  the 
data-reliability  survey,  we  also  reviewed  the  corresponding  database  user 
manual  and  related  documentation  to  determine  the  limitations  of  the 
data.  By  assessing  this  information  against  GAO  data-quality  standards, 
we  determined  that  the  data  presented  in  our  findings  were  sufficiently 
reliable  for  presenting  information  about  the  aircraft’s  reliability  in  this 
report. 

To  interview  officials  about  and  observe  F-35  operations,  maintenance, 
training,  and  developmental  and  operational  testing,  we  conducted  visits 
to  Eglin  Air  Force  Base,  Marine  Corps  Air  Station  Yuma,  and  Naval  Air 
Station  Patuxent  River — locations  where  these  activities  were  occurring. 

In  addition,  we  met  with  officials  about  F-35  sustainment  planning  and 
costs  at  Naval  Air  Systems  Command  and  we  reviewed  documentation 
and  interviewed  officials  at  Lockheed  Martin — the  prime  contractor — in 
Fort  Worth,  Texas,  about  the  program’s  status  with  respect  to 
development,  operations,  testing,  and  costs.  To  determine  the  extent  to 
which  DOD  has  developed  a  sustainment  strategy  and  addressed  key 
risks  related  to  affordability  and  operational  readiness,  we  reviewed 
DOD’s  guidance  and  policy  for  defense  acquisitions  and  life-cycle 
sustainment  planning  and  compared  them  to  the  program’s  Life  Cycle 
Sustainment  Plan  and  other  planning  documents,  including  business- 
case  analyses.  Furthermore,  we  reviewed  DOD’s  risk-management  and 
policy  guidance  for  specific  sustainment  elements  and  compared  it  to 
current  program  risks  and  mitigation  practices. 

To  determine  the  extent  to  which  DOD  has  developed  a  reliable  O&S  cost 
estimate  for  the  F-35  program,  we  evaluated  DOD’s  two  primary  cost 
estimates,  the  2013  JPO  and  the  2013  CAPE  office’s  O&S  cost 
estimates,  using  GAO’s  Cost  Estimating  and  Assessment  Guide.®  The 


"^Reliability  and  maintainability  data  measure  aircraft  performance  to  determine  how  often 
the  aircraft  experiences  failures  and  how  much  time  it  takes  to  repair  those  failures. 

®GAO,  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Deveioping  and 
Managing  Capitai  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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methodology  outlined  in  the  Cost  Estimating  and  Assessment  Guide  is  a 
compilation  of  best  practices  that  federal  cost-estimating  organizations 
and  industry  use  to  develop  and  maintain  reliable  cost  estimates 
throughout  the  life  of  an  acquisition  program.  The  JPO’s  estimate  is  used 
to  drive  budget  planning  and  is  required  by  DOD  acquisition  policy,  and 
CAPE’S  estimate  is  used  to  assess  affordability  and  is  required  at  major 
acquisition  milestones  or  by  special  request  from  DOD  leadership. 
Generally,  either  the  JPO’s  or  CAPE’s  estimate  is  used  as  DOD’s 
estimate  of  record  and  published  in  the  Selected  Acquisition  Report 
(SAR) — a  report  that  DOD  is  required  to  submit  annually  to  Congress  on 
each  of  its  major  defense  acquisition  programs.  For  the  F-35,  DOD 
decided  to  use  the  CAPE  O&S  estimate  in  the  2012  and  2013  SARs.  We 
evaluated  both  the  JPO  and  the  CAPE  O&S  estimates  because  both 
estimates  are  used  in  DOD  decision  making.  According  to  the  Cost 
Estimating  and  Assessment  Guide,  there  are  four  general  characteristics 
of  sound  cost  estimating:  being  well-documented,  comprehensive, 
accurate,  and  credible.  For  the  purposes  of  this  review,  we  conducted  a 
limited  assessment  and  evaluated  two  of  these  characteristics: 
comprehensive  and  credible.  We  chose  to  evaluate  the  estimates’ 
comprehensiveness  because  if  a  cost  estimate  is  not  comprehensive  then 
it  cannot  fully  meet  the  characteristics  of  being  well-documented  or 
accurate.  For  example,  if  the  cost  estimate  is  missing  some  cost 
elements,  then  the  documentation  will  be  incomplete  and  the  estimate  will 
be  inaccurate.  We  also  included  an  assessment  of  the  credible 
characteristic  due  to  the  substantial  estimated  cost  of  the  program  and  to 
determine  whether  risks  to  costs  were  quantified.  To  determine  whether 
the  comprehensive  and  credibility  characteristics  were  met,  we  reviewed 
CAPE  and  JPO  cost-estimating  documentation,  including  data  sources, 
assumptions,  and  calculations,  and  we  interviewed  cost-estimating 
officials  from  these  offices. 

We  conducted  this  performance  audit  from  August  2013  to  September 
2014  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  Appendix  I  provides  a 
more  detailed  description  of  our  scope  and  methodology. 


Page  4 


GAO-14-778  F-35  Sustainment 


Background 


The  F-35  Lightning  II  program  is  a  joint,  multinational  acquisition  intended 
to  develop  and  field  an  affordable  family  of  next-generation  strike  fighter 
aircraft  for  the  United  States  Air  Force,  Navy,  Marine  Corps,  and  eight 
international  partners.®  There  will  be  three  variants  of  the  F-35.  (1 )  The 
conventional  takeoff  and  landing  (CTOL)  variant,  designated  the  F-35A, 
will  be  a  multirole,  stealthy  strike  aircraft  replacement  for  the  Air  Force’s 
F-16  Falcon  and  the  A-10  Thunderbolt  II  aircraft,  and  will  complement  the 
F-22A  Raptor  (see  fig.  1 ).  (2)  The  short  takeoff  and  vertical  landing 
(STOVL)  variant,  the  F-35B,  will  be  a  multirole,  stealthy  strike  fighter  that 
will  replace  the  Marine  Corps’  F/A-18C/D  Hornet  and  AV-8B  Harrier 
aircraft.  (3)  The  carrier-suitable  variant  (CV),  the  F-35C,  will  provide  the 
Navy  a  multirole,  stealthy  strike  aircraft  to  complement  the  F/A-18  E/F 
Super  Hornet.  The  Marine  Corps  will  also  field  a  limited  number  of  F-35C 
CVs.  Lockheed  Martin  is  the  aircraft  contractor  and  Pratt  &  Whitney  is  the 
engine  contractor. 


Figure  1 :  F-35A  Air  Force  Conventionai  Takeoff  and  Landing  Variant  (CTOL) 


Source;  Air  Force.  |  GAO-14-778 


DOD  began  the  F-35  program  in  October  2001  with  a  highly  concurrent 
acquisition  strategy,  resulting  in  substantial  overlap  between 
development,  testing,  and  production.  The  program  was  restructured  in 
2003  and  again  in  2007  due  to  performance  problems,  cost  growth,  and 
schedule  slips.  After  the  program  experienced  a  critical  cost  breach  in 
2010,  the  program  was  restructured  once  more,  and  in  2012  some  near- 


®The  international  partners  are  the  United  Kingdom,  Italy,  the  Netherlands,  Turkey, 
Canada,  Australia,  Denmark,  and  Norway.  These  nations  contributed  funds  for  system 
development  and  signed  agreements  to  procure  aircraft.  In  addition,  Israel  and  Japan 
have  signed  on  as  foreign  military  sales  customers. 
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term  procurement  quantities  were  deferred  to  a  future  date  and  the 
program’s  cost  estimates  were  increased/  In  2013,  DOD  issued  a  report 
to  Congress®  stating  that  the  Marine  Corps,  Air  Force,  and  Navy  were 
planning  to  field  initial  operational  capabilities  (IOC)  in  2015,  2016,  and 
2018,  respectively/  See  figure  2  for  a  timeline  of  major  events  leading 
from  the  beginning  of  development. 


Figure  2:  Timeline  of  Major  Events  in  the  F-35  Program,  2001-2040 


Low-rate 

Operations  and 

production 

support  phase 

2001  I  2002  I  2003  |  2004  |  2005  2006  2007  2008  2009  |  2010  |  2011  |  2012  |  2013  |  2014  |  2015  |  2016  |  2017  |  2018  i  2019  {  2020-2040 


2001 :  Milestone  B  is  approved  and 
Lockheed  Martin  is  seiected  as  the 
prime  contractor.® 

The  program  begins  deveiopment. 


Source:  GAO  analysis  of  Department  of  Defense  information.  |  GAO-14-778 


2012:  Program 
estabiishes  new 
acquisition 
baseiine  and 
recertifies 
Milestone  B  after 
a  cost-growth 
breach. 


2019:  Planned 
milestone  decision 
to  enter  full-rate 
production. 

Navy  to  declare  IOC. 

begin  operational 
testing,  and  Marine 
Corps  to  declare 
Initial  Operational 
Capability  (IOC).® 


2016:  Air 
Force  to 
declare  IOC. 

2015:  Program  to  2018: 


'^As  part  of  DOD’s  acquisition  process,  Milestone  B  initiates  the  Engineering  and  Manufacturing 
Development  Phase. 

‘’IOC  is  generally  obtained  when  organizations  or  units  scheduled  to  receive  a  system  have  received 
it  and  have  the  ability  to  employ  and  maintain  that  system. 


^Section  2433  of  title  10  of  the  United  States  Code,  commonly  referred  to  as  Nunn- 
McCurdy,  requires  DOD  to  notify  Congress  whenever  a  major  defense  acquisition 
program’s  unit  cost  experiences  cost  growth  that  exceeds  certain  thresholds.  This  is 
commonly  referred  to  as  a  Nunn-McCurdy  breach.  Significant  breaches  occur  when  the 
program  acquisition  unit  cost  or  procurement  unit  cost  increases  by  at  least  15  percent 
over  the  current  baseline  estimate  or  at  least  30  percent  over  the  original  estimate.  For 
critical  breaches,  as  was  the  case  with  the  F-35  in  2010,  when  these  unit  costs  increase  at 
least  25  percent  over  the  current  baseline  estimate  or  at  least  50  percent  over  the  original, 
DOD  is  required  to  take  additional  steps,  including  conducting  an  in-depth  review  of  the 
program.  Programs  with  critical  breaches  must  be  terminated  unless  the  Secretary  of 
Defense  certifies  to  certain  facts  related  to  the  program  and  takes  other  actions,  including 
restructuring  the  program.  10  U.S.C.  §  2433a. 

®U.S.  Navy,  U.S.  Marine  Corps,  U.S.  Air  Force,  Report  to  Congressional  Defense 
Committees:  F-35  Initial  Operational  Capability  {June  2013). 

®IOC  is  generally  obtained  when  organizations  or  units  scheduled  to  receive  a  system 
have  received  it  and  have  the  ability  to  employ  and  maintain  that  system. 


Page  6 


GAO-14-778  F-35  Sustainment 


The  program  is  currently  in  low-rate  initial  production,  and  the  contractor 
is  assembling  the  next  lot  of  36  aircraft  with  a  scheduled  delivery  by  the 
end  of  2015.  As  of  June  2014,  78  aircraft  have  been  fielded,  are  flying, 
and  are  being  maintained  at  Eglin  Air  Force  Base,  Marine  Corps  Air 
Station  Yuma,  Edwards  Air  Force  Base,  Nellis  Air  Force  Base,  and  Luke 
Air  Force  Base.  The  program  had  achieved  about  18,250  cumulative  flight 
hours  for  the  fleet  of  aircraft;  a  total  of  200,000  flight  hours  are  required  in 
order  for  the  fleet  to  reach  R+M  maturity,  as  outlined  in  the  program’s 
Operational  Requirements  Document.  In  March  2014,  we  found  that  the 
program  had  progressed  in  its  developmental  flight  testing,  but  it 
continued  to  lag  in  testing  of  critical  mission-systems  software,  delaying 
the  delivery  of  expected  warfighting  capabilities.'"’ 

By  full-rate  production,  planned  for  fiscal  year  2019,  DOD  would  generally 
be  required  to  establish  adequate  sustainment  and  support  systems  for 
the  F-35.''''  Per  DOD  guidance  for  all  weapon  systems  acquisitions,  these 
sustainment  and  support  systems  should  be  defined  in  a  support  concept 
that  is  incorporated  into  a  sustainment  strategy.  For  the  F-35,  this  concept 
should  comprise  the  necessary  plans  to  conduct  operations, 
maintenance,  and  sustainment  throughout  the  system’s  life  cycle,  with  the 
F-35  Life  Cycle  Sustainment  Plan  serving  as  the  principle  document 
governing  F-35  sustainment.  According  to  the  F-35  Operational 
Requirements  Document,  this  concept  must  provide  warfighting  and 
peacetime  capability  with  the  lowest  cost  of  ownership,  and  all  variants 
must  be  able  to  deploy  rapidly,  sustain  high  mission  reliability,  and  sustain 
a  high  sortie-generation  rate.''^ 

In  developing  this  strategy,  the  program  has  made  affordability  its  top 
priority,  and  the  F-35  Program  Executive  Officer’^  reiterated  this  priority  in 


’°GAO-14-322. 

’’Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense  for  Acquisitions, 
Technology  and  Logistics,  Operation  of  the  Defense  Acquisition  System,  (Interim) 
Instruction  5000.02  (Washington,  D.C.:  Nov.  26,  2013). 

’^Sorties,  or  flights,  are  generated  in  support  of  testing  and  operations.  Department  of 
Defense,  Joint  Strike  Fighter,  Operationai  Requirements  Document  (Washington,  D.C.: 
Aug. 19,  2008). 

’^The  Program  Executive  Officer  position  alternates  between  the  Departments  of  the  Navy 
and  the  Air  Force,  and  reports  to  the  Service  Acquisition  Executive  of  the  other  service. 
The  Program  Executive  Officer  is  currently  an  Air  Force  Lieutenant  General. 
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his  April  2014  testimony. In  this  testimony,  the  Program  Executive 
Officer  also  identified  areas  in  which  the  program  is  focusing  on  driving 
down  sustainment  costs  and  identifying  improvements  to  meet  long-term 
sustainment  needs  in  order  to  “produce  a  mutually  beneficial  sustainment 
enterprise  that  operates,  manages,  and  supports  the  global  system  with 
relevant  metrics  and  incentives,  while  meeting  warfighter-defined 
readiness  and  cost  objectives.”  However,  the  department  has  stated  that 
it  has  a  long  history  of  starting  programs  that  proved  to  be  unaffordable, 
and  the  result  of  this  practice  has  been  costly  program  cancellations  and 
dramatic  reductions  in  inventory  objectives.^®  Recently  updated  DOD 
acquisition  policy  has  placed  more  emphasis  on  including  sustainment 
considerations  as  early  as  possible  in  the  acquisition  process.''® 

According  to  the  policy,  a  successful  program  meets  the  sustainment 
performance  requirements,  remains  affordable,  and  continues  to  seek 
cost  reductions  throughout  the  Operations  and  Support  Phase  that  begins 
after  the  full-rate  production  decision. 

In  a  2012  memorandum,  the  Under  Secretary  of  Defense  for  Acquisitions, 
Technology  and  Logistics  (AT&L)  acknowledged  that  certain  F-35 
program  areas  including  the  Autonomic  Logistics  Information  System 
(ALIS),  reliability  growth,  and  sustainment  costs  require  additional 
improvement. 

•  ALIS.  ALIS  is  the  primary  sustainment  tool  for  the  F-35  and  is 

intended  to  predict  maintenance  and  supply  issues,  automate  logistics 
support  processes,  and  provide  decision  aids  to  help  reduce  life-cycle 
sustainment  costs  and  improve  force  readiness.  ALIS  is  one  of  three 
major  components  that  make  up  the  F-35  air  system,  along  with  the 
aircraft  and  the  engine,  and  comprises  both  hardware  and  software. 
The  F-35  program  is  delivering  ALIS  capabilities  incrementally,  which 


^'^Fiscal  2015  Defense  Authorization:  Tacticai  Aircraft  Programs,  1 13th  Cong.  (April  2014) 
(statement  of  Lt.  Gen.  Christopher  C.  Bogdan,  Program  Executive  Officer,  F-35  Lightning 
II  Joint  Program  Office). 

^®DOD  Interim  Instruction  5000.02. 

^®DOD  Interim  Instruction  5000.02.  See  also  Defense  Acquisition  University,  Defense 
Acquisition  Guidebook  (Washington,  D.C.:  May  15,  2013).  The  guidebook  is  designed  to 
complement  policy  documents  such  as  DOD  Interim  Instruction  5000.02  by  providing  the 
acquisition  workforce  with  discretionary  best  practices  that  should  be  tailored  to  the  needs 
of  each  program. 
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include:  operations,  maintenance,  supply  chain,  customer-support 
services,  training,  tech  data,  system  security  and  external  interfaces. 

•  Reliability  and  Maintainability  (R+M).  R+M  measures  aircraft 
performance  to  determine  how  often  the  aircraft  experiences  failures 
and  how  much  time  it  takes  to  repair  those  failures.  R+M  is  monitored 
through  a  series  of  metrics  that  measure  the  intended  performance  of 
the  aircraft  in  meeting  its  requirements  by  maturity  at  a  cumulative 
200,000  flight  hours  with  at  least  50,000  flight  hours  per  variant.  R+M 
drives  sortie-generation  rates  and  a  reduced  logistics  footprint  for  the 
F-35,  as  well  as  informing  program  O&S  costs,  which  are  tied  to  the 
performance  of  the  system  at  maturity. 

•  O&S  Costs.  O&S  consists  of  sustainment  costs  incurred  from  the 
initial  system  deployment  through  the  end  of  system  operations  and 
includes  all  costs  of  operating,  maintaining,  and  supporting  a  fielded 
system.  The  F-35  JPO  develops  an  annual  estimate  for  the  O&S 
costs  of  maintaining  and  supporting  the  F-35  for  56  years. In  201 1 , 
the  estimate  was  $1.03  trillion;  in  2012,  it  was  $857  billion;  and  in 
2013,  it  was  $916  billion  in  then-year  dollars.  Additionally,  CAPE 
conducts  independent  cost  estimates  on  certain  programs  at  different 
points  in  the  acquisition  cycle.  In  recent  years,  CAPE’s  F-35  O&S  cost 
estimate  has  been  reported  in  DOD’s  annual  SAR  as  the  official  O&S 
cost  estimate  for  the  program.  In  2013,  that  estimate  was  about  $1 .02 
trillion  in  then-year  dollars. 

As  stated  earlier,  recent  changes  to  DOD’s  acquisition  guidance  and 
policy  have  also  focused  on  making  sustainment  decisions  earlier  in  the 
acquisition  process,  by  identifying  the  attributes  of  an  effective 
performance-based  logistics  arrangement  and  requiring  strategies  that 
identify  and  manage  technical  data  rights.  Both  of  these  areas  will  play  an 
important  role  in  the  future  of  the  F-35  program. 

•  Performance  Based  Logistics  (PBL).^®  An  effective  performance- 
based  logistics  strategy  ties  objective  metrics  for  delivered  logistical 
system  performance  to  incentives  that  will  motivate  the  support 
provider  (i.e.,  the  contractor).  In  recent  years,  F-35  program  managers 


^^Each  aircraft  has  a  30-year  service  life,  but  the  2013  estimates  span  56  years. 

1  ft 

PBL  is  performance-based  life-cycle  product  support  whereby  outcomes  are  acquired 
through  performance-based  arrangements  that  deliver  warfighter  requirements  and 
incentivize  product  support  providers  to  reduce  costs  through  innovation. 
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have  reaffirmed  their  commitment  to  employing  a  robust  performance- 
based  logistics  system  for  the  F-35  to  help  determine  options  for 
pursuing  the  most  value-driven  performance-based  agreements  with 
the  support  provider. 

•  Technical  Data  Rights.  Technical  data  for  weapon  systems  include 
the  details  necessary  to  ensure  the  adequacy  of  performance,  as  well 
as  instructions  for  operation,  maintenance,  and  other  actions  needed 
to  support  weapon  systems.''®  Technical  data  are  an  important  part  of 
a  weapon  system  program,  such  as  the  F-35.  Identifying  technical 
data  needs  and  costs  early  in  the  acquisition  process  can  help 
program  managers  maximize  the  potential  for  future  competition  and 
develop  the  system’s  overall  sustainment  strategy. 


DOD  Is  Developing 
an  F-35  Sustainment 
Strategy,  but  Key 
Risks  to  Affordability 
and  Operational 
Readiness  Remain 


DOD’s  development  of  an  F-35  sustainment  strategy  is  an  ongoing 
process  that  includes  a  number  of  plans,  analyses,  and  decisions  that 
must  be  finalized  by  the  start  of  full-rate  production — estimated  to  occur  in 
fiscal  year  2019.  To  date,  the  program’s  sustainment  strategy  continues 
to  evolve  but  may  not  be  affordable,  and  DOD  has  not  fully  addressed 
several  key  risks  to  long-term  affordability  and  operational  readiness,  that, 
if  not  mitigated,  could  affect  the  development  of  the  strategy. 


DOD  Is  Currently 
Developing  an  F-35 
Sustainment  Strategy,  but 
It  May  Not  Be  Affordable 


DOD  has  a  number  of  sustainment  planning  documents  that  will  make  up 
the  F-35  program’s  overall  sustainment  strategy,  including  the  Life  Cycle 
Sustainment  Plan,  which,  according  to  DOD  policy,  will  provide  the  basis 
for  sustainment  activities.  DOD  also  has  past  and  ongoing  studies  and 
analyses  that  have  the  potential  to  affect  and  inform  the  overall 
sustainment  strategy,  including  two  Business  Case  Analyses  and  several 
Level  of  Repair  Analyses.  These  planning  documents  and  analyses  will 
provide  the  basis  for  the  overall  sustainment  strategy  that  will  dictate 
sustainment  operations  when  the  program  enters  full-rate  production. 
Furthermore,  a  recently  launched  initiative  led  by  the  F-35  JPO  is  working 
to  establish  a  roadmap  to  the  end  of  the  design  phase  in  2018,  which  is 
intended  to  eventually  result  in  a  Future  Support  Construct — another 


^®See  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  clause  252.227- 
7013  for  a  full  definition  of  “technical  data.” 
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element  of  sustainment  planning  that  will  inform  decisions  for  long-term 
operations  and  support  for  the  aircraft.  The  JPO  plans  to  finalize  the 
Future  Support  Construct  in  March  2015. 

However,  the  current  sustainment  strategy  that  DOD  is  developing  may 
not  be  affordable.  The  program  has  continued  to  experience  cost 
overruns,  and  the  recent  SAR  estimated  the  O&S  cost  to  sustain  the 
system  for  56  years  to  be  approximately  $1  trillion.  Additionally,  according 
to  DOD  officials,  including  officials  within  the  Office  of  the  Secretary  of 
Defense  (OSD),  the  current  sustainment  strategy  is  not  affordable. 
According  to  CAPE  analysis,  the  combined  O&S  costs  of  several  legacy 
aircraft— the  F-15C/D,  F-16C/D,  AV-8B,  and  F-18A-D  fleets— in  2010 
exceeded  $1 1  billion.  Comparatively,  based  on  CAPE’s  2013  O&S  cost 
estimate,  the  annual  cost  to  sustain  the  F-35  will  be  about  $19.9  billion  (in 
base  year  2012  dollars)  in  2040 — the  end  of  its  steady-state  years. This 
$8.8  billion  difference  represents  an  increase  of  more  than  79  percent  in 
annual  O&S  costs  for  the  F-35  as  compared  with  several  legacy  aircraft 
(see  fig.  3).  Moreover,  the  Program  Executive  Officer  has  continued  to 
express  concerns  over  the  affordability  of  the  program’s  sustainment 
approach,  stating  that  “F-35  sustainment  costs  remain  a  concern”  and 
that  affordability  continues  to  be  a  top  priority  for  the  program.^'' 


For  the  purposes  of  this  report  we  define  steady-state  operations  as  the  period  from 
2036  to  2040,  when,  according  to  the  military  services’  plans,  the  number  of  F-35  aircraft 
and  flying  hours  reaches  its  highest  point  and  plateaus. 

Fiscal  2015  Defense  Authorization:  Tactical  Aircraft  Programs  (statement  of  Lt.  Gen. 
Christopher  C.  Bogdan). 
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Figure  3:  Comparison  of  Annuai  Estimated  F-35  Operating  and  Support  (O&S)  Cost 
at  Steady  State  to  Actuai  Legacy  Aircraft  O&S  Costs  in  Fiscai  Year  2010 


0  5  10  15  20 

Billions  of  doiiars 

Source;  GAO  presentation  of  Department  of  Defense  data  and  Air  Force  and  Marine  Corps  photos.  |  GAO-14-778 

Notes:  For  the  purposes  of  this  report,  we  define  steady-state  operations  as  the  period  from  2036  to 
2040,  when,  according  to  the  services’  plans,  the  number  of  F-35  aircraft  and  flying  hours  reaches  its 
highest  point  and  plateaus. 

^The  F-35  cost  presented  is  CAPE’S  estimated  total  annual  O&S  cost  for  2040  in  base  year  2012 
dollars. 

‘’Legacy  aircraft  cost  is  based  on  a  CAPE  analysis  of  2010  cost  data,  representing  a  high  point  for 
aircraft  O&S  budgets  due  to  contingency  operations  at  that  time. 


In  2012,  DOD  established  affordability  targets  for  the  program,  stating 
that  the  cost  per  flying  hour  for  the  Air  Force,  Marine  Corps,  and  Navy  F- 
35  variants  could  not  exceed  $35,200,  $38,400,  and  $36,300 
respectively.  However,  DOD  officials,  including  some  at  OSD,  stated  that 
they  believe  that  based  on  this  threshold,  the  program  is  not  affordable. 
To  address  increasing  costs,  DOD  has  taken  steps  focused  on  cost 
reduction  including,  but  not  limited  to,  establishing  a  Cost  War  Room  in 
2013  and  a  Readiness  Cell  in  2014.  The  Cost  War  Room  is  a 
collaborative  group  comprising  DOD,  Lockheed  Martin,  and  Pratt  & 
Whitney  personnel,  established  with  the  purpose  of  reducing  costs  and 
providing  support  for  transitioning  to  performance-based  logistics.  The 
mission  of  the  Readiness  Cell  is  to  identify  opportunities  to  enable  F-35 
availability  (a  sustainment  metric)  at  more  than  60  percent  by  2015 
across  all  three  variants,  thereby  improving  sustainment  metrics  that 
directly  affect  O&S  costs.  However,  it  is  unclear  whether  these  cost¬ 
saving  measures  can  result  in  an  affordable  approach  to  the  long-term 
operations  and  sustainment  of  the  F-35  because  the  established 
affordability  targets  that  must  be  achieved  may  not  be  representative  of 
what  the  services  can  actually  afford. 
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DOD’s  current  acquisition  policy  states  that  affordability  constraints  for 
procurement  and  sustainment  should  be  determined  early  in  program 
planning  processes  and  should  be  used  to  ensure  that  capability 
requirements  are  prioritized  and  that  cost  trade-offs  occur  as  early  as 
possible  and  throughout  the  program’s  life  cycle. These  constraints 
should  not  be  based  on  cost  estimates  but  rather  on  whether  a 
component  can  afford  the  estimated  costs  of  a  system,  and  should  be 
used  to  indicate  whether  actions  must  be  taken  to  prevent  exceeding  the 
constraints.^^  When  DOD  established  affordability  targets  for  the  F-35 
program  in  2012,  the  methodology  for  determining  that  threshold  was  not 
informed  by  actual  resource  constraints  within  service  budgets  at  the 
time.  Specifically,  DOD  officials  stated  that  the  targets  were  determined 
by  arbitrarily  lowering  CAPE’s  estimated  F-35  Cost  per  Flight  Hour  by  10 
percent. However,  without  informed  affordability  constraints,  based  on 
military  service  budgets,  DOD  cannot  be  sure  whether  the  costs  savings  it 
achieves  through  current  efforts  will  lead  to  an  affordable  sustainment 
strategy,  and  DOD  may  miss  additional  areas  for  savings. 


DOD  Has  Not  Taken 
Steps  to  Fully  Address 
Risks  to  Affordability  and 
Operational  Readiness 
That  Could  Affect  the  F-35 
Sustainment  Strategy 


DOD  faces  the  following  three  key  risks  to  the  long-term  affordability  and 
operational  readiness  of  the  F-35  program:  (1)  the  performance  of  ALIS; 
(2)  the  R+M  of  the  whole  air  system;  and  (3)  the  management  of  technical 
data  rights.  However,  the  department  has  not  taken  steps  to  fully  address 
these  risks,  which  could  affect  the  development  of  the  sustainment 
strategy. 


^^Affordability  goals  are  established  at  Milestone  A,  the  entry  into  Technology 
Development.  After  systems  engineering  trade-offs  are  completed  during  the  Technology 
Development  phase,  these  affordability  goals  then  become  affordability  caps  at  Milestone 
B,  the  start  of  System  Development,  when  a  match  is  to  be  made  between  requirements 
and  resources.  We  refer  to  the  goals  and  caps  collectively  as  affordability  constraints. 

^^DOD  Interim  Instruction  5000.02.  Prior  to  being  incorporated  into  the  interim  instruction, 
affordability  constraints  were  a  part  of  the  Better  Buying  Power  initiative. 

^^DOD  officials,  including  the  F-35  product  support  manager,  have  stated  that  linking 
affordability  to  actual  service  budgets  will  constitute  a  focal  point  of  the  program,  moving 
forward.  However,  to  date,  we  have  not  seen  any  documentation  or  evidence  indicating 
that  this  effort  is  in  progress. 
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ALIS 


ALIS  is  a  system  of  systems  that  serves  as  the  primary  logistics  tool  to 
support  operations,  mission  planning,  and  sustainment  of  the  F-35.  One 
official  described  ALIS  as  the  brains  of  the  aircraft,  helping  maintainers 
manage  such  things  as  aircraft  health  and  diagnostics,  supply-chain 
management,  and  necessary  maintenance  events.  However,  ALIS  has 
experienced  recurring  problems,  including  user  issues  and  schedule 
delays.  Furthermore,  DOD  has  stated  that  ALIS  is  one  of  the  biggest 
impediments  to  the  amount  of  sorties,  or  flights,  that  can  be  generated  in 
support  of  testing  and  operations.  The  sortie-generation  rate  is  a  key 
performance  parameter^®  for  the  aircraft. 

For  example,  the  integration  of  ALIS  capabilities — ^which  are  fielded  in 
increments — has  been  repeatedly  delayed.  The  most  recent  timeline  for 
ALIS  estimates  that  the  final  version  of  ALIS  will  be  released  for  testing  in 
2017 — 7  years  after  the  originally  planned  release  date.  Additionally, 
ALIS’s  diagnostic  system  has  not  reached  full  functionality.  Specifically,  it 
has  failed  to  meet  basic  requirements,  including  having  the  ability  to 
identify  faults  and  failures  in  the  aircraft.  That  issue  has  resulted  in  time- 
consuming  workarounds  that  could  further  affect  sortie  generation. 
Currently,  DOD  tracks  issues  with  ALIS  through  an  internal  reporting 
mechanism  that  allows  users  in  the  field  to  identify  problems  with  the 
system  for  potential  fixes.  For  example,  according  to  DOD  officials,  ALIS’s 
Anomaly  and  Failure  Resolution  System  (AFRS)  is  intended  to  identify 
the  necessary  maintenance  to  repair  the  aircraft,  but  it  is  not  yet  mature. 
Maintainers  at  an  F-35  operational  training  site  told  us  that,  as  a  result, 
they  have  had  to  use  multiple  approaches  to  identify  the  best 
maintenance  solution.  Once  identified,  the  maintainer  can  submit  this 
solution  as  an  update  for  AFRS.  However,  this  update  must  first  be 
reviewed  by  field  support  on  site  and  then  sent — in  the  form  of  an  action 
request — to  the  contractor  for  approval  before  it  is  integrated  into  AFRS. 
Maintenance  officials  told  us  that  they  have  submitted  several  thousand 
action  requests  to  date  and  have  thereby  created  a  backlog,  leaving 
maintainers  to  wait  multiple  days  for  an  approval. 

Problems  with  ALIS  are  reported  by  users  on  an  individual  basis,  and 
thus  do  not  give  DOD  a  complete  view  of  ALIS’s  system  performance. 
Once  an  issue  is  reported,  DOD  decides  whether  to  (1)  defer  the  issue. 


key  performance  parameter  (KPP)  is  a  performance  attribute  of  a  system  considered 
critical  to  the  development  of  an  effective  military  capability. 
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forcing  a  workaround  for  users,  or  (2)  pay  for  the  contractor  to  develop  a 
“fix.”  For  an  overview  of  this  process,  see  figure  4. 


Figure  4:  Autonomic  Logistics  Information  System  (ALIS)  Issue-Resolution  Process 


Sample  of  ALIS  capabilities 


Diagnostic  heaith  codes  are  downioaded 
from  the  piane  to  Autonomic  Logistics 
Information  System  (ALIS). 


Solution 

identified 


Contractor 
receives  issue 
reports  from 
operations. 


Solution  sent 
back  to 

maintainer  and 
ALIS  is  updated. 


Maintainer 
retrieves  codes 
from  ALIS  and 
reports  any  issues 
to  the  contractor. 


•  Assess  usage 

•  Perform  aircraft  maintenance 

•  Perform  air  vehicle  post-  and 
preflight  servicing 

•  Plan  mission 

•  Process  diagnostics  data 

•  Process  sustainment  data 

•  Provide  logistics  data 


Source:  GAO  analysis  of  Department  of  Defense  information;  Lockheed  Martin.  [  GAO-14-778 

^If  a  solution  is  not  immediately  identified  by  the  contractor,  a  temporary  solution  is  sent  back  to  the 
maintainer,  or  a  workaround  is  used,  or  both,  until  a  permanent  solution  is  identified. 


To  fully  understand  system  performance,  DOD’s  Systems  Engineering 
Guide  for  Systems  of  Systems  (SOS)  states  that  it  is  important  to  have  a 
set  of  metrics  that  allow  for  an  assessment  of  the  SOS  performance 
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traced  to  user  requirements  because  the  SOS  will  likely  evolve  based  on 
incremental  changes  in  individual  systems — similar  to  ALIS’s  incremental 
fielding. These  metrics  should  measure  the  intended  integrated 
behavior  and  performance  of  the  SOS  in  actual  operations  versus  the 
progress  of  the  development  of  the  SOS,  allowing  an  assessment  of  SOS 
capabilities  based  on  user  requirements,  such  as  functionality  between 
interdependent  subsystems.  Additionally,  according  to  the  F-35 
Operational  Requirements  Document,  the  F-35  must  have  a  fully 
functional  and  effective  logistics  system  (e.g.,  ALIS)  aimed  at  ensuring 
operational  readiness  and  availability  and  reducing  overall  life-cycle  costs 
by  the  time  the  aircraft  reaches  maturity  at  200,000  flight  hours  for  all 
three  variants  with  at  least  50,000  per  variant. 

Although  DOD  has  a  performance-measurement  process  for  the  overall 
aircraft,  it  does  not  have  a  similar  process,  with  metrics  and  targets,  to 
determine  and  address  the  most  significant  performance  issues  with 
ALIS.  DOD  receives  data  on  ALIS’s  availability  from  Lockheed  Martin, 
and  it  tracks  the  progress  of  development  of  ALIS,  but  it  does  not  track 
any  other  data  to  assess  the  performance  or  maturity  of  the  system  based 
on  user  requirements.  Without  a  performance-measurement  process  that 
ties  ALIS  performance  to  user  requirements,  it  is  unclear  how  the 
program  can  effectively  and  efficiently  address  future  ALIS  performance 
issues.  Having  performance  metrics  and  targets  for  ALIS  that  assess  the 
intended  integrated  behavior  and  performance  in  actual  operations  could 
allow  the  program  to  determine  future  root  causes  of  ALIS  issues — a 
major  tenet  of  risk  mitigation,  according  to  DOD’s  risk  management 
guide — and  address  them.^^  Furthermore,  because  the  current  process  to 
address  ALIS  issues  involves  paying  for  ALIS  fixes  on  an  incremental 
basis,  the  program  may  face  additional  costs  that  are  more  difficult  to 
predict.  Moreover,  without  a  performance-measurement  process  for  ALIS, 
DOD  cannot  be  sure  that  it  is  considering  the  most  important 
performance-related  information  when  developing  future  performance- 
based  logistics  agreements  for  the  program — agreements  that  will  affect 
long-term  sustainment  costs.  Finally,  without  a  fully  functional  logistics 
system  (i.e.,  ALIS),  DOD  may  not  be  able  to  fully  support  the  necessary 


^®The  System  Engineering  Guide  for  Systems  of  Systems  provides  practitioners  with 
practical  guidance  and  is  intended  as  reference  only. 

^^Department  of  Defense,  Risk  Management  Guide  for  Acquisition  (Washington,  D.C.: 
August  2006). 
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Reliability  and  Maintainability 


performance  parameters,  such  as  sortie  generation,  to  ensure  operational 
readiness  and  availability. 

DOD  officials  have  continued  to  express  concerns  with  the  reliability  and 
maintainability  (R+M)  of  the  aircraft.  In  his  April  2014  testimony,  the 
Program  Executive  Officer  stated  that  R+M  remains  an  area  for  needed 
improvement.  In  that  same  month,  the  Director,  Operational  Test  and 
Evaluation  (DOT&E),  stated  that  the  growth  rates  and  improvement 
programs  that  would  be  necessary  to  meet  R+M  requirements  when  the 
aircraft  reaches  maturity  (200,000  flight  hours)  are  what  they  termed 
“ambitious.”  Additionally,  DOT&E  stated  that  future  efforts  to  improve 
R+M  may  be  hampered  by  higher  failure  rates  as  the  aircraft  begins  to  fly 
with  more  aggressive  maneuvering  and  more  extensive  use  of  mission 
systems. 

To  measure  the  R+M  of  the  aircraft,  testers  collect  data  on  (1)  the  number 
of  flight  hours  achieved  before  a  failure  occurs  (reliability)  and  (2)  the 
amount  of  time  it  takes  to  repair  those  failures  (maintainability).  These  two 
primary  measures  are  supported  by  various  metrics  and  tracked  along 
planned  growth  curves  to  measure  progress  in  meeting  requirements.^® 
Based  on  recent  growth  curves  and  reports  for  some  of  DOD’s  R+M 
metrics,  some  metrics  have  progressed,  some  continued  to  lag,  and 
some  have  worsened.  For  example: 

•  Mean  Flight  Hours  between  Failures  (Design  Controllable)  is  the 

average  amount  of  flight  hours  achieved  before  a  design-controllable 
failure^®  occurs.  As  of  March  2014,  this  metric  was  progressing  in  that 
the  number  of  flight  hours  before  a  failure  occurs  was  increasing  for 
all  three  variants.  For  example,  the  average  flight  hours  between 
failures  for  the  F-35A — the  variant  with  the  most  flight  hours  to  date — 
was  5.2  in  March  2014,  surpassing  the  expectation  at  its  current  flight 
hours  by  about  1.2  and  growing  toward  its  requirement  at  maturity  of 
6.0.  Moreover,  this  was  an  increase  of  about  1 .8  average  flight  hours 
between  failures  since  September  2013,  as  reported  by  GAO.®° 


oo 

Each  R+M  metric  must  meet  its  specific  requirement  at  maturity — 200,000  flight  hours 
for  the  fleet,  with  at  least  50,000  per  variant. 

^®Design-controllable  failures  are  those  that  can  be  attributed  to  deficiencies  in  design,  but 
considered  by  the  JPO  to  be  fixable  by  design  modification. 

®°GAO-14-322. 
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•  Mean  Flight  Hours  between  Critical  Failures  is  the  average  amount 
of  flight  hours  achieved  before  a  failure  occurs  that  results  in  the  loss 
of  a  capability  to  perform  a  mission-essential  function.^''  As  of  March 
2014,  this  metric  was  lagging  well  below  its  requirements  at  maturity, 
meeting  an  average  of  42  percent  of  those  requirements  across  all 
three  variants. 

•  Mean  Time  to  Repair  is  the  average  time  it  takes  a  maintainer  to 
repair  a  failed  component  or  device.  Currently,  this  metric  is  not 
improving  in  that  as  flight  hours  increase,  it  is  taking  maintainers 
longer  to  repair  failed  components  for  the  F-35A  and  F-35C,  and  the 
amount  of  time  it  takes  to  repair  failed  components  for  the  F-35B 
remains  unchanged.  Specifically,  GAO  reviewed  R+M  growth  curves 
provided  by  DOD  showing  the  historical  growth  of  this  metric  from 
2009  for  the  F-35B  and  201 0  for  the  F-35A  and  F-35C  to  March  2014, 
and  we  observed  that  the  metric  is  trending  in  the  opposite  direction  of 
its  predicted  path  for  the  F-35A  and  F-35C,  and  the  metric  is 
remaining  steady,  without  improvement,  for  the  F-35B. 

A  tenet  of  F-35  sustainment  is  to  continually  improve  R+M  of  the  weapon 
system  to  drive  down  O&S  costs.  In  an  effort  to  improve  R+M,  DOD 
began  a  Reliability  Improvement  Plan  in  2014  that  focused  on  addressing 
the  top  20  issues  with  the  aircraft  that  affect  R+M.  While  the  current 
Reliability  Improvement  Plan  is  in  the  process  of  addressing  some  R+M 
issues,  the  plan  focuses  primarily  on  the  aircraft’s  hardware  and  does  not 
specifically  address  software  and  its  potential  effects  on  overall  R+M.  The 
F-35  is  the  most  software-intensive  fighter  aircraft  DOD  has  procured  to 
date.  According  to  DOD  officials,  software  reliability  is  an  issue  that  needs 
attention;  however,  officials  have  also  stated  that  the  current  process  for 
tracking  R+M  is  almost  exclusively  dedicated  to  hardware. 

To  identify  some  software  issues  as  they  arise,  users  in  the  field  use  an 
internal  system  to  submit  requests  to  the  contractor,  but  these  requests 
are  submitted  on  an  individual  basis  and  may  not  always  be  addressed 
immediately  as  it  takes  time  to  determine  whether  the  issue  is  related  to 
hardware  or  software.  For  example,  officials  told  us  that  the  Electrical 
Optical  Targeting  System,  which  is  used  to  track  a  target,  continues  to 
fail.  In  this  instance,  testers  reported  the  problem,  and  officials  attempted 
to  improve  the  capability  with  hardware  changes.  However,  not  all  issues 


O'! 

The  loss  of  a  mission-essential  function  also  downs  the  aircraft  for  that  mission. 
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with  the  Electrical  Optical  Targeting  System  were  fixed  with  the  hardware 
changes,  and  officials  have  decided  to  also  try  to  address  the  issue  with 
software  changes,  causing  users  to  identify  workarounds  in  the 
meantime.  As  another  example,  officials  discussed  instances  in  which  the 
diagnostics  system  signals  to  a  maintainor  that  the  landing  gear  failed, 
but  it  was  actually  a  sensor  near  the  landing  gear  that  failed.  Because 
software  for  isolating  these  types  of  failures  is  not  yet  mature,  operators 
and  maintainors  on  the  ground  may  continue  to  check  the  landing  gear 
without  discovering  the  sensor  issue  and  reporting  it.  Finally,  officials 
stated  that  with  the  release  of  the  next  increment  of  software  in  2015,  a 
number  of  new  issues  may  arise,  and  these  issues  may  be  related  to 
software  because  the  new  software  includes  new  processors  that  can 
affect  mission  systems  on  the  aircraft. 

According  to  DOD  reliability  guidance,  basic  reliability  activities  should 
include  software  reliability  assessments.^^  Additionally,  Naval  Air  Systems 
Command  guidance  advises  that  weapon  systems  programs  should 
establish  software  R+M  metrics  as  a  key  activity  in  terms  of  achieving 
supportability  goals  and  developing  a  high  level  of  confidence  that  the 
product  will  work  reliably  and  can  be  maintained  easily. Furthermore, 
DOD’s  Risk  Management  Guide  states  that  effective  risk  management 
depends,  in  part,  on  continuous  monitoring  and  reassessment  to  identify 
and  address  root  causes  of  program  risks. Currently  there  are  no 
processes  or  metrics  by  which  the  program  can  monitor  and  assess 
software  R+M  and  the  full  extent  to  which  software  issues  may  be 
contributing  to  the  overall  R+M  issues  of  the  aircraft.  Without  an 
assessment  process,  including  metrics,  for  software  R+M  as  a  means  of 
mitigating  ongoing  risks  to  R+M,  program  officials  cannot  have  a  full 
understanding  of  the  root  causes  of  R+M  issues  on  the  aircraft,  or 
whether  software  failures  contribute  to  the  overall  R+M  outside  of 
hardware  failures.  This  can  potentially  lead  to  unforeseen  costs  in  the 
future  as  software  is  continually  upgraded  and  refined.  Since  O&S  costs 


^^Department  of  Defense,  Handbook:  Reliability  Growth  Management  (Washington,  D.C.: 
Sept.  10,  2009).  This  handbook  is  for  guidance  only  and  is  not  to  be  cited  as  a 
requirement. 

oo 

Naval  Air  Systems  Command,  Software  Logistics  Pr/mer  (Washington,  D.C.:  April  2010). 
This  primer  is  for  training  purposes  only. 

^^DOD,  Risk  Management  Guide  for  Acquisition.  This  guidance  is  not  mandatory  to  follow, 
but  program  managers  are  encouraged  to  apply  the  fundamentals  of  the  guide. 
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Technical  Data 


are  based  on  meeting  the  required  reliability  at  maturity  (200,000  flight 
hours),  there  are  increasing  risks  to  O&S  cost  and  future  aircraft 
availability. 

In  May  201 1 ,  we  found^®  that  DOD  needs  access  to  technical  data — 
recorded  information  used  to  produce,  support,  maintain,  or  operate  a 
system^® — related  to  its  weapon  systems  in  order  to  help  control  costs 
and  maintain  flexibility  in  the  acquisition  and  sustainment  of  those 
weapon  systems.  Technical  data  can  enable  the  government  to  complete 
maintenance  work  in-house,  as  well  as  to  competitively  award  acquisition 
and  sustainment  contracts.  Additionally,  we  found  that  for  service 
contracts  pertaining  to  DOD  weapon  programs,  which  can  involve 
products  as  well  as  support  services,  the  lack  of  access  to  proprietary 
technical  data  and  a  heavy  reliance  on  specific  contractors  for  expertise 
creates  limitations  to,  or  even  precludes,  the  possibility  of  competition.®^ 

Within  the  past  5  years,  acquisition  reforms  such  as  the  Weapon  Systems 
Acquisition  Reform  Act  of  2009  and  other  initiatives  seeking  greater 
efficiency  and  cost  savings  in  acquisitions  have  put  greater  emphasis  on 
obtaining  technical-data  rights  and  on  maintaining  competition  throughout 
the  life  cycle  of  weapon  systems.  In  2010,  DOD  announced  its  “Better 
Buying  Power”  initiative  that  outlines  a  series  of  actions  and  directives  to 
promote  competition,  including  ensuring  that  technical  data  requirements 


®®GAO,  Defense  Acquisition:  DOD  Shouid  Ciarify  Requirements  for  Assessing  and 
Documenting  Technicai-Data  Needs,  GAO-11-469  (Washington,  D.C.:  May  11, 2011). 

®®Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  clause  252.227-7013 
defines  technical  data  as  “recorded  information,  regardless  of  the  form  of  method  of  the 
recording,  of  a  scientific  or  technical  nature  (including  computer  software  documentation) 
...  [but  not  including]  computer  software  or  data  incidental  to  contract  administration,  such 
as  financial  and/or  management  information.”  Technical  data  for  weapon  systems  include 
drawings,  specifications,  standards,  and  other  details  necessary  to  ensure  the  adequacy 
of  item  performance,  as  well  as  manuals  that  contain  instructions  for  installation, 
operation,  maintenance,  and  other  actions  needed  to  support  weapon  systems. 

®^GAO,  Federai  Contracting:  Opportunities  Exist  to  increase  Competition  and  Assess 
Reasons  When  Oniy  One  Offer  is  Received,  GAO-1 0-833  (Washington,  D.C.:  July  26, 
2010). 
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are  considered  at  key  milestones  of  the  acquisition  cycle. Specifically,  in 
November  2010,  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  issued  a  memorandum  that  requires  program 
officials  to  (1)  conduct  a  business  case  analysis  that  outlines  the  technical 
data  rights  the  government  will  pursue  to  ensure  competition,  and  (2) 
include  the  results  of  this  analysis  in  certain  strategies  at  Milestone  B.^® 
However,  according  to  DOD  officials,  because  the  F-35  program’s  initial 
Milestone  B  decision  occurred  back  in  2001,  and  these  initiatives  started 
nearly  a  decade  later,  the  F-35  program  is  getting  a  late  start  on  its 
assessment  of  its  technical  data  needs. 

In  accordance  with  the  Acquisition  Strategy  for  the  October  2001 
Milestone  B,  the  F-35  program  did  not  acquire  technical  data  suitable  for 
competition  for  the  F-35  or  its  subsystems  under  its  competitively 
awarded  system  development  contract.  Furthermore,  as  stated  in  the 
Acquisition  Strategy,  since  the  system  design  contract  is  in  place,  the 
acquisition  of  data  suitable  for  competition  may  be  prohibitively  expensive 
and  would  not  result  in  a  commensurate  return  on  investment.  The 
current  iteration  of  the  Acquisition  Strategy  leaves  the  door  open  for 
acquiring  current  and  future  technical  data  that  could  support  future 
competition  through  the  life  cycle.  Specifically,  DOD  plans  to  assess  the 
merits  of  acquiring  operations,  maintenance,  installation,  and  training  data 
that  would  support  sustainment  and  operation  efforts.  Planning  for  follow- 
on  procurements  is  to  include  data  rights/management  strategies  that  will 
support,  to  the  extent  practical,  the  options  for  competition  through  the 
remainder  of  the  program  life  cycle.  With  respect  to  this  assessment,  and 
DOD’s  current  requirements  related  to  technical  data,  DOD  officials 
confirmed  the  need  for  the  F-35  program  to  begin  identifying  technical 
data  needs  and  costs  related  to  future  sustainment  of  the  aircraft.  They 


®®Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  “Better  Buying 
Power:  Mandate  for  Restoring  Affordability  and  Productivity  in  Defense  Spending,” 
memorandum  (June  28,  2010);  “Better  Buying  Power:  Guidance  for  Obtaining  Greater 
Efficiency  and  Productivity  in  Defense  Spending,”  memorandum  (Sept.  14,  2010); 
“Implementation  Directive  for  Better  Buying  Power — Obtaining  Greater  Efficiency  and 
Productivity  in  Defense  Spending,”  memorandum  (Nov.  3,  2010);  and  “Better  Buying 
Power  2.0:  Continuing  the  Pursuit  for  Greater  Efficiency  and  Productivity  in  Defense 
Spending,”  memorandum  (Nov.  13,  2012). 

oq 

As  part  of  DOD’s  acquisition  process.  Milestone  B  initiates  the  Engineering  and 
Manufacturing  Development  phase  and  is  normally  the  formal  initiation  of  an  acquisition 
program.  DOD  Interim  Instruction  5000.02.  Milestone  B  for  the  F-35  was  reached  in  2001, 
and  recertified  in  2012. 
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said  that  DOD  has  begun  discussions  with  contractor  officials  to 
determine  who,  pursuant  to  the  applicable  contract  clauses,  has  the  rights 
to  specific  technical  data  associated  with  the  various  hardware  and 
software  systems.  According  to  JPO  officials,  technical-data  rights  could 
play  a  significant  role  in  the  development  of  the  sustainment  strategy  for 
the  F-35;  however,  the  federal  government  does  not  have  an 
understanding  of  the  technical  data  it  currently  owns,  what  technical  data 
it  may  still  need,  and  how  much  it  may  cost  to  acquire  those  data  to 
support  the  future  sustainment  of  the  F-35. 

According  to  current  DOD  acquisition  policy,  an  Intellectual  Property  (IP) 
Strategy  must  be  established  and  maintained  for  all  defense  acquisition 
programs  in  order  to  identify  and  manage  the  full  spectrum  of  IP  and 
related  issues,  such  as  technical  data,  from  the  inception  of  the  program 
and  throughout  the  life  cycle.  The  IP  Strategy  describes  how  program 
management  will  assess  program  needs  for,  and  acquire  competitively 
whenever  possible,  the  IP  deliverables  and  associated  license  rights 
necessary  for  competitive  and  affordable  acquisition  and  sustainment 
over  the  entire  product  life  cycle.  The  IP  Strategy  is  to  be  updated 
throughout  the  entire  product  life  cycle,  summarized  in  the  Acquisition 
Strategy,  and  presented  with  the  Life  Cycle  Sustainment  Plan  during  the 
Operations  and  Support  Phase. The  IP  Strategy  is  also  to  integrate,  for 
all  systems,  the  IP  planning  elements  required  under  Subpart  207.106  (S- 
70)  of  the  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS). 

The  DFARS"*^  specifically  states  the  following: 

•  In  accordance  with  Section  802(a)  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2007  (Pub.  L.  No.  109-364)  and 
DOD  policy  requirements,  acquisition  plans'^^  for  major  weapon 
systems  and  subsystems  of  major  weapon  systems  shall — 


^°DOD  Interim  Instruction  5000.02. 

^^DOD,  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  Subpart  207.106 
(S-70).  The  DFARS  provides  DOD  implementation  and  supplementation  of  the  Federal 
Acquisition  Regulation  (FAR).  The  DFARS  contains  requirements  of  law,  DOD-wide 
policies,  delegations  of  FAR  authorities,  deviations  from  FAR  requirements,  and  policies 
and  procedures  that  have  a  significant  effect  beyond  the  internal  operating  procedures  of 
DOD  or  a  significant  cost  or  administrative  effect  on  contractors.  DFARS  §  201 .301 . 

^^Acquisition  plans  are  generally  developed  before  issuance  of  a  solicitation,  FAR  §  7.104 
and  DFARS  §  207.106. 
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•  assess  the  long-term  technical  data  and  computer  software  needs 
of  those  systems  and  subsystems;  and 

•  establish  acquisition  strategies  that  provide  for  the  technical  data 
and  computer  software  deliverables  and  associated  license  rights 
needed  to  sustain  those  systems  and  subsystems  over  their  life 
cycle;  the  strategy  may  include 

•  the  development  of  maintenance  capabilities  within  DOD;  or 

•  competition  for  contracts  for  sustainment  of  the  systems  or 
subsystems. 

According  to  the  current  F-35  acquisition  strategy,  the  F-35  program  will 
assess  the  merits  of  acquiring  current  and  future  technical  data  that  could 
support  future  competition  throughout  the  life  cycle,  and  will  help  provide 
the  basis  for  identifying  the  data  and  data  rights  required  for  the 
sustainment  of  the  program.  However,  as  of  July  2014,  DOD  officials  had 
not  completed  assessments  of  technical  data  rights,  including  a  plan  that 
would  identify  what  technical  data  rights  the  federal  government  currently 
owns,  what  additional  technical  data  rights  it  may  need,  and  the  costs 
associated  with  purchasing  any  necessary  technical  data.  According  to 
DOD  officials,  the  program  is  planning  to  address  these  technical  data 
rights  issues  as  part  of  the  future  sustainment  strategy  that  will  document 
decisions  for  long-term  operations  and  support  for  the  aircraft.  To  date, 
although  DOD  officials  have  acknowledged  the  importance  of  technical 
data  rights  to  the  future  of  the  F-35  program  and  have  told  us  that 
discussions  with  contractor  officials  regarding  ownership  of  those  rights 
have  begun,  we  have  not  seen  any  documentation  confirming  that 
assessments  of  technical  data  rights  will  be  completed.  Until  DOD 
determines  the  technical  data  rights  the  federal  government  currently 
owns  and  its  critical  technical  data  needs  and  associated  costs,  the 
program  runs  the  risk  of  being  limited  in  its  flexibility  to  make  changes  to 
sustainment  plans  and  the  overarching  sustainment  strategy. 

Furthermore,  without  a  long-term  IP  Strategy,  the  program  runs  the  risk  of 
not  being  able  to  compete  requirements  for  such  things  as  spare  parts. 
Unless  technical  data  rights  needs  are  considered  up  front,  critical  data 
and  software  may  not  be  acquired,  rendering  them  unavailable  (or 
unaffordable)  years  later  when  seeking  to  maximize  competition  on  a 
program  during  its  sustainment  phase. 
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Improved 
Assumptions  and 
Additional  Analyses 
Could  Increase 
Reliability  of  F-35 
O&S  Costs 


DOD’s  most  recent  O&S  cost  estimates  are  comprehensive  in  that  they 
include  all  DOD-required  program  elements  and  are  organized  according 
to  a  standard  O&S  cost-estimating  structure;  but  weaknesses  exist  with 
respect  to  a  few  of  the  assumptions,  and  the  estimates  did  not  include  all 
analyses  necessary  to  make  them  fully  reliable/^ 


DOD  Created  Two  Primary 
Estimates  of  Operating 
and  Support  Costs  for  the 
F-35  across  Its  Life  Cycle 


There  are  two  primary  DOD  F-35  O&S  cost  estimates,  one  developed  by 
the  JPO  and  one  by  CAPE.  O&S  costs  include  the  direct  and  indirect 
costs  of  sustaining  a  fielded  system,  such  as  costs  for  replacement  parts, 
fuel,  maintenance,  personnel,  support  facilities,  and  equipment  (see  table 
1).  The  JPO’s  2013  estimate,  which  is  used  to  drive  budget  planning,  lists 
the  total  F-35  O&S  costs  across  a  56-year  life  cycle  as  approximately 
$916  billion  in  then-year  dollars.  CAPE’s  estimate,  which  is  used  to 
assess  affordability  and  is  required  at  major  milestones  or  by  special 
request,  lists  the  same  cost  to  be  about  $1.02  trillion  in  then-year 
dollars. Generally,  either  the  JPO’s  or  CAPE’s  estimate  is  used  as 
DOD’s  estimate  of  record  and  published  in  the  Selected  Acquisition 
Report  (SAR) — a  report  that  DOD  is  required  to  submit  annually  to 
Congress  on  each  of  its  major  defense  acquisition  programs.'^® 


table  scoring  the  cost  estimates  based  on  these  characteristics  and  their 
corresponding  best  practices  can  be  found  in  app.  II. 

4^10  U.S.C.  §  2334.  For  the  F-35,  DOD  decided  to  use  the  CAPE  estimate  for  the  2012 
and  2013  SARs.  Flowever,  both  estimates  are  used  in  DOD  decision  making,  as  noted 
above.  Consequently,  we  evaluated  both  the  JPO  and  the  CAPE  estimates.  Total  O&S 
costs  are  listed  in  then-year  dollars,  which  are  adjusted  to  show  the  effects  of  inflation 
over  the  life  cycle  of  the  program. 

^®10  U.S.C.  §  2432.  A  full  life-cycle  cost  analysis,  including  O&S  costs,  is  required 
annually  in  the  program’s  Selected  Acquisition  Report  (SAR).  Generally,  the  program 
office  estimate  is  used  as  the  estimate  of  record  and  published  in  the  SAR,  and,  as  a 
result,  the  JPO  estimate  is  developed  annually. 
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Table  1:  JPO  and  CAPE  2013  Total  Estimated  O&S  Costs 

Then-year  dollars  in  billions 

Operating  and  Support  (O&S)  cost  element 

Joint  Program  Office 
(JPO) 

Cost  Assessment  and  Program 
Evaluation  (CAPE) 

Mission  personnel 

$212.9 

$236.4 

Unit-level  consumption® 

167.9 

158.7 

Maintenance 

270.6 

326.3 

Unit-level  maintenance® 

246.8 

282.9 

Intermediate-level  maintenance® 

0.0 

0.0 

Depot-level  maintenance 

23.8 

43.4 

Contractor  and  sustaining  support® 

158.8 

190.5 

Indirect  support® 

106.0 

104.6 

Total 

$916.3 

$1,020.0 

Source:  GAO  analysis  of  JPO  and  CAPE  data.  |  GAO-14-778 


Notes:  Both  life-cycle  cost  estimates  were  developed  in  2013  and  span  56  years — the  JPO  estimate 
from  2009  to  2065  and  the  CAPE  estimate  from  2010  to  2066.  Figures  in  table  may  not  sum  due  to 
rounding. 

^Consolidates  fuel,  training  expendables,  and  other  unit-level  consumption. 

‘’Consolidates  unit-level  consumables,  parts,  and  depot-level  repairables,  which  include  repairable 
individual  parts,  assemblies,  or  subassemblies  that  are  required  for  the  repair  of  the  aircraft  and 
related  equipment. 

‘’Intermediate-level  maintenance  represents  the  cost  of  labor  and  materiel  for  maintenance  that  is  not 
performed  at  the  operational  or  depot  levels,  but  by  an  intermediate-level  maintenance  organization. 
Intermediate-level  maintenance  activities  may  include  calibration,  repair,  testing,  and  replacement  of 
parts,  components,  or  assemblies,  and  technical  assistance. 

‘‘Consolidates  sustaining  support  and  contractor  support  from  CAPE  estimate  and  sustaining  support 
and  continuing  system  improvements  from  JPC  estimate.  Sustaining  support  relates  to  the  cost  of 
support  activities  other  than  maintenance  that  can  be  attributed  to  a  system  and  are  provided  by 
organizations  other  than  operating  units.  Continuing  system  improvements  relate  to  the  cost  of 
hardware  and  software  modifications  to  keep  the  system  operating  and  current. 

'’Indirect  support  represents  the  cost  related  to  general  services,  such  as  medical  care  for  active-duty 
military  and  base  operating  support. 


DOD  F-35  Cost  Estimates 
Are  Comprehensive,  but 
Weaknesses  Exist  in  a 
Few  Assumptions 


The  JPO  and  CAPE  both  created  comprehensive  F-35  O&S  cost 
estimates,  but  there  are  weaknesses  in  a  few  of  the  assumptions  used  in 
the  estimates.  According  to  the  GAO  Cost  Estimating  and  Assessment 
Guide,  a  good  cost  estimate  depends  on  how  well  the  program  is  defined 
in  the  technical  baseline  and  work-breakdown  structure.  Both  the  JPO 
and  CAPE  used  a  DOD-approved  technical  baseline  and  a  product- 
oriented  O&S  cost  work-breakdown  structure  to  develop  their  2013 
estimates.  The  technical  baseline  provides  the  comprehensive  program 
description,  which  is  used  to  determine  what  must  be  included  in  the 
estimate.  A  work-breakdown  structure  is  a  necessary  program- 
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management  tool  because  it  provides  a  basic  framework  for  estimating 
costs,  allowing  for  the  estimates  to  be  easily  updated  and  compared  to 
future  estimates.  Both  DOD  guidance  and  the  GAO  Cost  Estimating  and 
Assessment  Guide  highlight  the  importance  of  good  documentation  in 
cost  estimating,  stating  that  a  complete  cost  estimate  should  be  formally 
documented  to  serve  as  an  audit  trail  of  source  data,  methods,  and 
results."^®  In  addition,  the  GAO  Cost  Estimating  and  Assessment  Guide 
states  that  all  assumptions  and  exclusions  the  estimate  is  based  on 
should  be  reasonable,  clearly  identified,  and  explained. While  the  JPO 
and  CAPE  estimates  are  comprehensive,  weaknesses  exist  because  both 
estimates  used  unreasonable  assumptions  for  fuel  burn  rates  and  have 
not  clearly  documented  intermediate-level  maintenance  costs  (see  app.  II 
for  more  information  on  our  analysis). 

•  Fuel  burn  rates:  The  JPO  and  CAPE  estimates  do  not  use  a 

reasonable  assumption  for  the  Marine  Corps  fuel  burn  rate  across  the 
life  cycle  of  the  aircraft.  Both  estimates  use  a  fuel  burn  assumption  of 
1 ,493  gallons  per  flying  hour  for  the  Marine  Corps  variant.  While  the 
rate  used  in  the  estimates  was  calculated  based  on  service-planned 
missions  at  maturity  and  DOD-validated  physics-based  models,  this 
burn  rate  is  an  almost  exact  match  of  the  burn  rate  being  observed  at 
the  operational  sites.  This  may  be  a  reasonable  assumption  for  the 
short  term,  but  it  is  likely  that  the  fuel  burn  rates  will  increase  in  the 
long  term.  The  burn  rate  of  1,493  gallons  per  flying  hour  used  in  the 
JPC  and  CAPE  cost  estimates  reflects  fuel  burn  data  from  aircraft 
flown  using  limited  capabilities — at  slower  speeds  and  at  lower 
altitudes  than  the  F-35  will  eventually  fly.  After  2015,  the  Marine  Corps 
is  planning  on  using  the  aircraft’s  increased  capability,  which  will  likely 
result  in  more  fuel  being  used.  In  addition,  shortly  after  2015,  the 
Marine  Corps  is  planning  on  deploying  the  F-35  to  ships,  where  the 
aircraft’s  fuel-intensive  vertical  landing  capability  will  be  used  more 
frequently,  which  will  likely  increase  fuel  usage.  Consequently,  the 
use  of  lower  fuel  burn  rate  across  the  entire  life  cycle  of  the  Marine 
Corps  variant  is  not  a  reasonable  assumption. 


'’^®Office  of  the  Secretary  of  Defense,  Cost  Analysis  Improvement  Group,  Operating  and 
Support  Cost  Estimating  Guide  (Washington,  D.C.:  October  2007)  and  GAO-09-3SP.  The 
Cost  Analysis  and  Improvement  Group  is  now  known  as  Cost  Assessment  and  Program 
Evaluation  (CAPE). 

^^GAO-09-3SP.  We  are  defining  a  reasonable  assumption  as  one  that  is  logical,  credible, 
and  acceptable  given  the  available  data  and  information. 
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Similarly,  the  JPO  estimate  lowered  its  fuel  burn  rate  assumption  to 
1 ,480  gallons  per  flight  hour  from  1 ,558  gallons  per  flight  hour  for  the 
Air  Force  variant.  However,  the  Air  Force  stated  that  the  more 
conservative  assumption  of  1 ,558  gallons  per  flight  hour  should  be 
used  across  the  life  cycle  because  the  F-35  has  yet  to  use  its  full  flight 
capabilities,  weapons,  or  mission  systems,  which  will  likely  increase 
the  fuel  burn  rates  in  the  long  term.  The  higher  fuel  burn  rate 
assumption  for  the  Air  Force  would  represent  a  $4.0  billion  cost 
increase  in  base  year  2012  dollars  across  the  life  cycle  of  the  aircraft. 

•  Intermediate-level  maintenance.  These  costs  are  not  clearly 

documented  in  either  the  JPO  or  the  CAPE  cost  estimates.  For  the  F- 
35,  intermediate-level  maintenance  tasks  include  wheel  and  tire 
servicing,  battery  maintenance,  structural  airframe  support,  and 
inspections.  The  documentation  supporting  DOD’s  cost  estimate 
states  that  intermediate-level  maintenance  costs  are  not  calculated  for 
the  F-35,  but  the  program’s  maintenance  planning  documents  state 
that  intermediate-level  maintenance  will  be  used  in  the  sustainment  of 
the  F-35,  and  it  is  currently  being  employed.  Specifically,  Marine 
Corps  officials  told  us  that  intermediate-level  personnel  at  Yuma  are 
conducting  some  F-35  activities,  including  tire,  wheel,  and  battery 
servicing.  In  addition  to  the  existing  F-35  intermediate-level 
maintenance  being  conducted  at  Yuma,  DOD  has  conducted  at  least 
five  analyses  to  date,  each  resulting  in  a  recommendation  to  repair  a 
F-35  Marine  Corps  variant  component  at  the  intermediate  level.  These 
five  component  repairs  represent  more  than  $30  million  in 
intermediate-level  maintenance  costs  across  the  life  cycle  of  the 
aircraft,  but  they  are  not  documented  in  DOD’s  cost  estimates. 
Moreover,  all  of  the  analyses  to  identify  which  components  will  be 
repaired  at  the  intermediate  level  are  not  yet  complete,  and  the  costs 
related  to  intermediate-level  maintenance  could  increase.  According 
to  DOD  officials,  the  costs  of  these  intermediate-level  repairs  are 
already  included  under  mission  personnel  and  unit-level  maintenance 
elements  of  the  cost  estimates.  However,  this  is  not  documented  in 
the  mission  personnel  or  unit-level  maintenance  sections  of  the  cost 
estimates.  While  DOD  guidance  states  that  intermediate-level  costs 
can  be  combined  with  unit-level  costs,  the  guidance  also  states  that 
this  should  be  noted  in  the  cost  estimate’s  supporting  documentation 
to  avoid  an  interpretation  that  a  portion  of  the  maintenance  costs  were 
omitted.  Without  clear  documentation  of  these  costs  it  is  unclear 
whether  they  are  fully  represented  in  the  JPO  and  CAPE  estimate. 

In  addition,  the  JPO  estimate  does  not  include  reasonable  assumptions 

about  part-replacement  rates  and  depot  maintenance. 
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•  Part  replacement:  The  JPO  estimate  does  not  include  reasonable 
assumptions  for  part  replacement.  Based  on  data  from  the  Air  Force 
and  Marine  Corps  F-35  variants  at  testing  and  operational  sites,  parts 
are  being  replaced,  on  average,  15  to  16  times  more  frequently  than 
the  assumptions  used  across  the  life  cycle  of  the  JPO  estimate  (see 
table  2).  For  example,  a  sensor  that  costs  about  $4,800  is  being 
replaced  60  to  129  times  more  frequently  than  anticipated  across  the 
life  cycle  of  the  JPO  cost  estimate.  Another  example  is  the  battery 
charger  unit,  which  costs  about  $60,000  to  acquire  new,  and  is  being 
replaced  3  to  8  times  more  frequently  than  anticipated  across  the  life 
cycle  of  the  JPO  cost  estimate. 


Table  2:  Sample  of  F-35  Parts  with  Higher  Removal  Rates  Than  Assumed  in  Joint  Program  Office  (JPO)  2013  Cost  Estimate 

Air  Force  variant: 
number  of  times  more 

Marine  Corps  variant: 
number  of  times  more 

frequently  removed  than 

frequently  removed  than 

Replacement  cost 

Component/part 

assumed  by  JPO 

assumed  by  JPO 

per  part  (dollars) 

Thermal  management  system  fan 

10 

22 

$17,732 

Fiber  channel  switch 

13 

9 

33,485 

Data  security  module 

13 

12 

239,256 

270  volt  battery  unit 

1® 

2 

132,717 

Data  transfer  cartridge 

4 

5 

20,477 

Helmet  display  management  computer 

1® 

3 

209,266 

Signal  processor 

6 

6 

125,095 

Oxygen  generation  system 

2 

8 

41,985 

Seat  portion  assembly 

5 

3 

52,628 

Hot  air  sensor 

60 

129 

4,797 

General  processor 

6 

3 

85,648 

Helmet  display  unit 

3 

5 

124,917 

Pump  module 

19 

13 

36,660 

270  volt  battery  charger  unit 

3 

8 

59,745 

Distribution  valve 

192 

83 

6,314 

Panoramic  cockpit  display  electronics  unit 

2 

1 

146,179 

Integrated  power  package  ventilation  fan 

6 

9 

4,574 

Average  of  all  195  Air  Force  and  189 
Marine  Corps  variant  parts  analyzed’’ 

16 

15 

$47,624 

Source:  GAO  analysis  of  JPO  data.  |  GAO-14-778 


Notes:  The  parts  presented  in  this  table  are  taken  from  a  JPO  list  of  “high  driver”  replacement  parts 
from  November  2013,  some  of  which  will  be  addressed  in  the  JPO’s  ongoing  reliability-improvement 
efforts.  The  mean  flight  hours  between  removals  (MFHBR)  data  used  in  this  analysis  were  taken  from 
operational  and  test  Air  Force  F-35A  (CTOL)  and  Marine  Corps  F-35B  (STOVL)  variants  during  a  12- 
month  period  ending  on  March  31 , 2014.  The  JPO  assumptions  for  removals  are  developed  by  the 
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contractor  and  represent  the  predicted  frequency  of  removals  (MFHBR)  once  the  aircraft  reaches 
maturity,  which  is  when  the  entire  F-35  fleet  reaches  a  cumulative  200,000  flying  hours.  Navy  variant 
data  were  excluded  from  this  analysis  because  of  the  limited  number  of  Navy  variant  aircraft  being 
maintained  to  date. 

®These  frequencies  are  slightly  higher  than  one,  but  have  been  rounded  down. 

‘’This  average  includes  the  17  “high  driver”  replacement  parts  listed  above. 


The  part-replacement  assumptions  used  by  the  JPO  reflect  the 
anticipated  reliability  of  the  aircraft  at  maturity — once  the  entire  fleet 
has  achieved  200,000  flight  hours.  According  to  JPO  officials,  the 
reliability  issues  causing  the  high  part-replacement  rates  will  be 
resolved  once  the  aircraft  reaches  maturity,  which  is  estimated  to 
occur  at  the  end  of  fiscal  year  2019.  The  JPO  increased  the  cost  of 
replacing  parts  in  the  2010  to  2019  portion  of  its  estimate  to  reflect  the 
lower  reliability  of  the  aircraft  until  maturity."^®  However,  according  to 
officials  from  the  Institute  for  Defense  Analysis,  who  conducted  a 
study  of  the  F-35’s  R+M  for  DOT&E,  the  F-35  program  would  have  to 
achieve  a  higher  reliability-growth  improvement  rate  than  has  been 
observed  in  almost  all  other  aircraft  in  order  to  meet  the  anticipated 
reliability  by  2020."^®  As  previously  stated,  reliability  improvement 
efforts  are  underway  that  could  reduce  these  costs,  but  it  is  unlikely 
that  these  efforts  will  bring  significant  results  in  the  near  term  because 
the  current  F-35  fleet  must  be  modified  into  the  configuration 
necessary  for  reliability  improvement.  As  a  result,  for  the  next  10 
years,  the  F-35  fleet  will  not  represent  the  configurations  necessary 
for  reliability-growth  improvement.  In  addition,  according  to  the  study 
of  the  F-35’s  R+M  conducted  by  the  Institute  for  Defense  Analyses  for 
DOT&E,  future  efforts  to  increase  reliability  may  be  hampered  by 
increased  failure  rates  as  the  fielded  fleet  starts  flying  more 
operationally  representative  missions,  with  more  aggressive 
maneuvering  and  more  extensive  use  of  mission  systems. 

CAPE  adjusted  its  part-replacement  assumptions  for  its  2013  estimate 
to  reflect  higher  replacement  rates  across  the  life  cycle  of  the  aircraft. 


4ft 

The  JPO  increased  the  cost  of  replacement  parts  from  201 0  to  201 9  of  the  estimate  by 
using  a  reliability  growth  curve.  This  reliability  growth  curve  roughly  doubles  the 
replacement  part  cost  per  flight  hour  at  maturity  and  then  decreases  the  cost  on  an  annual 
basis,  assuming  that  parts  will  be  replaced  less  frequently  until  the  aircraft  reaches  the 
contractor-predicted  replacement  rate  at  maturity  in  2020. 

According  to  Institute  for  Defense  Analysis  officials,  this  rate  of  improvement  is  not 
impossible,  but  has  only  been  observed  in  dissimilar  aircraft  like  the  C-17. 
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As  a  result,  CAPE’s  2013  estimated  cost  for  parts  across  the  life  cycle 
of  the  program  is  $23  billion  greater  than  the  JPO’s  in  base  year  2012 
dollars.  CAPE  officials  told  us  that  if  they  had  used  the  actual 
replacement  rates  being  observed  at  the  F-35  sites,  the  part- 
replacement  cost  across  the  life  cycle  of  the  program  would  have 
been  roughly  $120  billion  greater  than  the  JPO’s  in  base  year  2012 
dollars. 

•  Depot  maintenance:  Finally,  the  JPO  estimate  does  not  include  a 
reasonable  assumption  regarding  F-35  depot  maintenance.  The  depot 
maintenance  cost  element  includes  the  labor,  materiel,  and  overhead 
costs  for  overhaul  or  rework  of  aircraft  returned  to  a  government  or 
contractor  repair  facility.  This  O&S  cost  includes  scheduled  and 
unscheduled  depot  maintenance,  as  well  as  inspections.  We  have 
found  this  to  be  a  cost  element  that  has  been  underestimated  in  the 
past  for  legacy  and  fifth-generation  aircraft,  including  the  F/A-18  and 
F-22.®°  The  JPO  reduced  its  assumptions  related  to  the  cost  of  depot 
maintenance  by  about  half,  from  about  $2  million  per  scheduled  depot 
induction  in  the  2012  estimate  to  about  $600,000-$1 .3  million  in  the 
2013  estimate.  This  change  resulted  in  a  $19.7  billion  reduction  in 
then-year  dollars  from  the  JPO’s  2012  estimate.  JPO  officials  stated 
that  this  reduction  was  the  result  of  using  a  different  cost-estimation 
method,  which  had  the  effect  of  lowering  the  hourly  cost  per  induction. 
According  to  the  assumptions  in  the  JPO’s  2013  estimate,  the 
average  cost  per  depot  induction  hour  is  roughly  $205.  However,  in 
2004,  the  average  cost  per  depot  induction  was  about  $238,  and  it  is 
likely  that  labor  and  materiel  costs  have  increased  over  the  past  10 
years. To  develop  its  hourly  cost,  the  JPO  used  F-16  data  from 
Lockheed  Martin  contractors  that  had  been  adjusted  for  the  F-35. 
According  to  the  JPO’s  current  assumptions,  materiel  costs  would  be 
30  percent  of  the  labor  costs,  but  data  for  the  AV-8  and  F/A-18  depot 
inductions  used  in  the  previous  year’s  estimate  indicate  that  depot 
induction  materiel  costs  have  historically  been  closer  to  45  percent  of 


®°GAO,  Defense  Management:  DOD  Needs  Better  Information  and  Guidance  to  More 
Effectively  Manage  and  Reduce  Operating  and  Support  Costs  of  Major  Weapon  Systems, 
GAO-1 0-71 7  (Washington,  D.C.:  July  20,  2010). 

^^This  average  price  per  depot  hour  includes  labor,  materiel,  and  overhead  costs  based 
on  data  from  three  air  logistics  centers — the  Oklahoma  City  Air  Logistics  Center,  Tinker  Air 
Force  Base,  Oklahoma;  the  Ogden  Air  Logistics  Center,  Hill  Air  Force  Base,  Utah;  and  the 
Warner  Robins  Air  Logistics  Center,  Robins  Air  Force  Base,  Georgia. 
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labor  costs.  As  a  result,  it  is  likely  that  the  depot  maintenance  hourly 
cost  used  in  the  JPO’s  2013  estimate  is  not  a  reasonable  assumption. 


DOD  Has  Not  Conducted 
Uncertainty  Analyses  to 
Determine  Credibility  of 
Estimates 


Every  cost  estimate  contains  a  degree  of  uncertainty  because  of  the 
many  assumptions  that  must  be  made  about  the  future.  To  mitigate  this 
uncertainty,  a  variety  of  checks  and  analyses  can  be  conducted  to 
determine  the  credibility  of  the  assumptions  and  the  estimate  as  a  whole 
While  the  JPO  and  CAPE  have  done  some  cross-checks  and  sensitivity 
analyses  to  mitigate  the  uncertainty  of  their  assumptions,  they  have  not 
conducted  uncertainty  analyses  to  determine  whether  the  estimates 
reflect  the  most  likely  costs. 


According  to  the  GAO  Cost  Estimate  and  Assessment  Guide  and  DOD 
guidance  on  cost  estimating,  a  good  practice  for  determining  the 
reasonableness  of  an  estimate’s  assumptions  for  certain  cost  elements  is 
to  cross-check  them  with  other  programs’  estimates  or  other  cost- 
estimate  methodologies  to  determine  whether  results  are  similar.  JPO 
and  CAPE  cost  estimators  stated  that  they  cross-checked  their  results  of 
key  cost  elements  against  O&S  estimates  for  other  aircraft  to  determine 
whether  they  produced  similar  results.  For  example,  according  to  a  JPO 
cost-estimating  official,  the  F-35  results  for  the  depot  maintenance  cost 
element  were  three  to  five  times  greater  than  the  same  costs  for  legacy 
aircraft.  In  addition,  CAPE  conducted  a  cross-check  comparing  the  cost 
per  flying  hour  of  legacy  and  F-35  aircraft,  which  identified  F-35  costs  as 
being  significantly  higher  than  the  legacy  aircraft.  The  official  stated  that 
this  was  reasonable  given  the  complexity  of  the  fifth-generation  aircraft 
and  the  additional  capability  that  the  F-35  offers. 


The  JPO  and  CAPE  have  also  conducted  some  sensitivity  analyses,  but 
they  have  not  conducted  uncertainty  analyses  to  determine  the  credibility 
of  their  estimates.  Sensitivity  analyses  examine  how  changes  to  individual 
assumptions  and  inputs  affect  the  estimate  as  a  whole.  Both  the  JPO  and 
CAPE  cost  estimators  have  told  us  that  they  conducted  “what  if 
scenarios — varying  the  assumptions  of  certain  variables  individually  to 
see  how  the  total  O&S  costs  may  be  affected.  Although  CAPE  did  not 
conduct  a  sensitivity  analysis  on  its  2013  O&S  cost  estimate,  it  did 
conduct  one  for  its  201 1  F-35  O&S  cost  estimate,  which  contains  many  of 
the  same  assumptions  as  its  2013  estimate.  However,  as  a  best  practice, 
a  sensitivity  analysis  should  be  included  in  all  cost  estimates  because  it 
examines  the  effects  of  changing  assumptions  and  ground  rules.  The 
JPO  asked  officials  at  the  Office  of  the  Secretary  of  Defense,  Logistics 
and  Materiel  Readiness  (OSD-LMR),  to  conduct  a  sensitivity  analysis 
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using  the  JPO  cost  model.  OSD-LMR  conducted  this  analysis, 
determining  the  cost  elements  that  represented  the  highest  percentage  of 
cost  and  varying  assumptions  to  determine  which  factors  have  the 
greatest  effect  on  the  total  O&S  cost.  According  to  DOD  guidance, 
sensitivity  analyses  are  useful  for  identifying  critical  estimating 
assumptions,  but  they  have  limited  utility  in  providing  a  comprehensive 
sense  of  overall  uncertainty.  In  contrast,  an  uncertainty  or  quantitative  risk 
analysis  can  provide  a  broad  overall  assessment  of  variability  in  the  cost 
estimate.  Similarly,  the  GAO  Cost  Estimating  and  Assessment  Guide 
states  that  risk  and  uncertainty  analyses  should  be  performed  to 
determine  the  level  of  risk  associated  with  the  estimate.  JPO  and  CAPE 
cost  estimators  have  not  conducted  uncertainty  analyses  to  understand 
the  potential  range  of  costs  around  their  point  estimates  and  to  convey 
their  level  of  confidence  that  the  estimates  reflect  the  most  likely  costs. 
JPO  and  CAPE  officials  stated  that  they  did  not  conduct  uncertainty 
analyses  because  they  are  not  a  standard  practice  for  DOD  when 
creating  O&S  cost  estimates.  However,  according  to  the  GAO  Cost 
Estimating  and  Assessment  Guide,  an  estimate  without  a  risk  and 
uncertainty  analysis  is  unrealistic  because  it  does  not  assess  the 
variability  in  the  estimate. 

In  April  2014  testimony,  the  F-35  program  executive  officer  highlighted 
the  importance  of  refining  the  cost  estimate  as  a  key  component  of  DOD’s 
risk-reduction  efforts  related  to  the  F-35.^^  Without  fully  reliable  O&S  cost 
information  to  develop  accurate  budget  requests  and  address  the 
affordability  of  the  F-35  program,  the  military  services  may  be  at  risk  of 
sustainment  cost  overruns  in  the  near  and  long  terms.  The  use  of 
improved  assumptions,  as  well  as  an  understanding  of  the  level  of 
confidence  in  the  estimates  through  the  use  of  an  uncertainty  analysis, 
would  also  improve  DOD’s  ability  to  make  F-35  sustainment  decisions. 


Conclusions 


The  F-35  program  remains  one  of  the  top  priorities  for  the  future  of  our 
national  defense,  according  to  senior  defense  leadership.  However,  as 
we  have  previously  reported,  the  F-35  program  has  experienced  cost, 
schedule,  and  performance  problems  that  have  put  it  7  years  behind 
schedule  and  significantly  over  its  original  budget.  With  life-cycle 


Fiscal  2015  Defense  Authorization:  Tactical  Aircraft  Programs  (statement  of  Lt.  Gen. 
Christopher  C.  Bogdan). 
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sustainment  costs  estimated  by  DOD  at  about  $1  trillion,  the  F-35  is 
approaching  a  critical  juncture.  In  an  era  of  significant  budgetary 
pressures  and  competition  for  resources,  with  the  Air  Force,  Navy,  and 
Marine  Corps  each  beginning  to  deploy  and  maintain  F-35  aircraft  within 
the  next  4  years,  and  with  full-rate  production  planned  in  fiscal  year  2019, 
DOD  must  find  ways  to  make  the  sustainment  of  the  F-35  program 
affordable.  DOD  has  begun  to  devote  attention  to  sustainment  through 
the  creation  of  plans  and  analyses  that  could  help  inform  programmatic 
decisions  and  shape  its  overall  sustainment  strategy.  One  key  decision  is 
for  DOD  to  determine  what  it  can  realistically  afford  with  respect  to  the  F- 
35  program,  but  until  it  identifies  affordability  constraints  tied  to  the 
military  services’  budgets,  it  will  continue  to  develop  and  field  the  most 
costly  weapon  system  program  in  history  without  knowing  whether  the  Air 
Force,  Navy,  or  Marine  Corps  can  pay  for  it.  Furthermore,  while  DOD  has 
begun  testing  and  fielding  the  F-35  at  sites  around  the  country,  the 
program  faces  several  risks,  including  ALIS,  software  R+M,  and 
uncertainties  related  to  technical  data  rights,  which  could  adversely  affect 
DOD’s  sustainment  strategy.  If  these  risks  remain  unaddressed, 
sustainment  costs  could  potentially  increase  well  beyond  current 
estimated  levels,  and  operational  readiness  could  suffer. 

Finally,  although  DOD’s  current  cost  estimates  identify  most  sustainment 
costs  and  document  cost-influencing  assumptions,  certain  elements  may 
be  underestimated.  In  some  areas,  such  as  the  assumptions  underlying 
future  fuel  burn  rates,  continuing  to  underestimate  these  elements  could 
translate  to  billions  of  dollars  in  cost  increases  over  the  life  cycle  of  the 
program.  Therefore,  it  is  imperative  that  DOD  provide  an  accurate  picture 
of  the  current  and  future  state  of  the  program,  as  well  as  a  greater  degree 
of  assurance  that  its  cost  estimates  are  credible — through  the  use  of 
uncertainty  analyses — to  help  inform  the  sustainment  strategy.  With  more 
than  2,400  F-35  aircraft  still  planned  for  purchase,  DOD  has  an 
opportunity  to  chart  a  more  informed,  analytical,  and  affordable  path  for 
the  program.  However,  sustainment  traditionally  represents  about  70 
percent  of  a  weapon  system’s  life-cycle  cost,  and  until  DOD  develops  a 
viable  sustainment  strategy,  addresses  key  risks,  and  improves  its  cost 
estimates,  the  department  will  likely  continue  to  present  Congress  with 
near-term  funding  requests  without  having  full  knowledge  of  the  extent  of 
the  program’s  long-term  financial  requirements. 
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Recommendations  for 
Executive  Action 


We  are  making  the  following  eight  recommendations  to  improve  DOD’s 
sustainment  planning  and  operating  and  support  (O&S)  cost  estimates  for 
the  F-35  program. 

To  help  DOD  develop  an  affordable  sustainment  strategy  for  the  F-35,  we 
recommend  that  the  Secretary  of  Defense  direct  the  Under  Secretary  of 
Defense  for  Acquisitions,  Technology  and  Logistics  to  direct  the  F-35 
Program  Executive  Officer  to  establish  affordability  constraints  linked  to, 
and  informed  by,  military  service  budgets  that  will  help  guide  sustainment 
decisions,  prioritize  requirements,  and  identify  additional  areas  for 
savings  by  March  2015,  at  which  point  the  Future  Support  Construct 
decision  will  be  approved. 

To  help  DOD  address  key  risks  to  F-35  affordability  and  operational 
readiness,  and  to  improve  the  reliability  of  its  O&S  cost  estimates  for  the 
life  cycle  of  the  program,  we  recommend  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to  take  the  following  five 
actions: 

•  To  enable  DOD  to  better  identify,  address,  and  mitigate  performance 
issues  with  the  Autonomic  Logistics  Information  System  (ALIS)  that 
could  have  an  effect  on  affordability,  as  well  as  readiness,  establish  a 
performance-measurement  process  for  ALIS  that  includes,  but  is  not 
limited  to,  performance  metrics  and  targets  that  (1)  are  based  on 
intended  behavior  of  the  system  in  actual  operations  and  (2)  tie 
system  performance  to  user  requirements. 

•  To  develop  a  high  level  of  confidence  that  the  aircraft  will  achieve  its 
R+M  goals,  develop  a  software  reliability  and  maintainability  (R+M) 
assessment  process,  with  metrics,  by  which  the  program  can  monitor 
and  determine  the  effect  that  software  issues  may  have  on  overall  F- 
35  R+M  issues. 

•  To  promote  competition,  address  affordability,  and  inform  its 
overarching  sustainment  strategy,  develop  a  long-term  Intellectual 
Property  (IP)  Strategy  to  include,  but  not  be  limited  to,  the 
identification  of 

•  current  levels  of  technical  data  rights  ownership  by  the  federal 
government  and 

•  all  critical  technical  data  needs  and  their  associated  costs. 

•  To  improve  the  reliability  of  the  JPO  F-35  O&S  cost  estimate,  clearly 
document  assumptions  related  to  intermediate-level  maintenance 
costs  and  revise  assumptions  related  to  fuel  burn  rates,  part 


Page  34 


GAO-14-778  F-35  Sustainment 


replacement,  and  depot-maintenance  induction  in  its  future  F-35  O&S 
cost  estimates  to  better  reflect  the  current  and  future  state  of  the  F-35 
program. 

•  To  understand  the  potential  range  of  costs  associated  with  the  JPO  F- 
35  O&S  cost  estimate,  conduct  uncertainty  analyses  on  future  JPO 
estimates. 

To  improve  the  reliability  of  the  Cost  Assessment  and  Program  Evaluation 
(CAPE)  F-35  O&S  cost  estimate,  we  recommend  that  the  Secretary  of 
Defense  direct  the  Director  of  CAPE  to  take  the  following  two  actions  for 
future  F-35  O&S  cost  estimates: 

•  clearly  document  assumptions  related  to  intermediate-level 
maintenance  and  revise  fuel  burn  assumptions  to  better  reflect  the 
current  and  future  state  of  the  F-35  program  and 

•  conduct  uncertainty  analyses  to  understand  the  potential  range  of 
costs  associated  with  its  estimates  to  reflect  the  most  likely  costs 
associated  with  the  program. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  concurred  with  seven 
of  the  report’s  eight  recommendations  and  partially  concurred  with  one 
recommendation.  DOD’s  comments  are  summarized  below  and  reprinted 
in  appendix  III.  DOD  also  provided  technical  comments,  which  we  have 
incorporated  into  our  report  where  appropriate. 

DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  Under  Secretary  of  Defense  for  Acquisitions,  Technology,  and 
Logistics  to  direct  the  F-35  Program  Executive  Officer  to  establish 
affordability  constraints  linked  to,  and  informed  by,  military  service 
budgets.  DOD  stated  that  it  established  F-35  affordability  targets  for  unit 
recurring  flyaway  cost  and  for  sustainment  cost  in  March  2012  to  guide 
development  of  an  affordable  strategy.  DOD  further  stated  that  the  F-35 
program  is  currently  operating  under  these  affordability  targets  and  using 
affordability  goals  to  reduce  life-cycle  costs.  We  acknowledge  that  the 
department  established  affordability  targets  for  sustainment  in  March 
2012,  but  these  actions  do  not  fully  address  the  intent  of  our 
recommendation  because,  as  we  note  in  our  report,  these  affordability 
targets  may  not  be  representative  of  what  the  services  can  actually  afford 
because  the  methodology  for  determining  the  targets  was  not  informed  by 
resource  constraints  within  military  service  budgets.  We  further 
acknowledge  in  our  report  that  DOD  is  currently  engaged  in  cost- 
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reduction  efforts,  but  until  DOD  establishes  affordability  targets  that  are 
informed  by  military  service  budgets,  the  department  cannot  be  sure 
whether  the  cost  savings  it  achieves  through  current  efforts  will  lead  to  an 
affordable  sustainment  strategy.  If  DOD  develops  affordability  targets 
that  are  linked  to  and  driven  by  military  service  budgets,  this  action  should 
address  the  recommendation. 

DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to  establish  a  performance- 
measurement  process  for  ALIS  that  includes,  but  is  not  limited  to, 
performance  metrics  and  targets  that  (1)  are  based  on  intended  behavior 
of  the  system  in  actual  operations  and  (2)  tie  system  performance  to  user 
requirements.  DOD  stated  that  the  F-35  program  will  continue  to  use 
ALIS  performance  metrics  and  targets  to  assess  effects  on  the 
operational  requirements,  such  as  the  sortie-generation  rate  key 
performance  parameter,  through  the  remainder  of  system  development 
and  demonstration  (SDD).  DOD  also  stated  that  while  the  F-35  program 
continues  to  assess  these  effects  today,  future  incremental  ALIS 
functional  deliveries  and  additional  operational  experience  will  enable 
more-refined  metrics  over  time  which  will  help  reduce  performance  risk 
and  ensure  a  successful  operational  test  and  evaluation  at  the  end  of 
SDD.  As  our  report  states,  we  acknowledge  that  metrics  for  ALIS’s 
availability  are  used  to  track  the  development  of  ALIS,  but  these  metrics 
do  not  fully  address  the  intent  of  our  recommendation  because  there  is  no 
performance-measurement  process  that  directly  ties  ALIS  performance  to 
user  requirements.  While  the  sortie-generation  rate  key  performance 
parameter  is  one  way  in  which  to  monitor  performance,  this  metric  is 
aimed  at  measuring  the  performance  of  the  entire  F-35  air  system  and 
not,  specifically,  how  ALIS  performs  its  functions  in  response  to  user 
requirements.  We  agree  that,  as  DOD  noted,  performance  metrics  and 
targets  may  be  refined  over  time  to  help  reduce  performance  risk,  but 
until  DOD  develops  performance  targets  and  metrics  tied  directly  to  ALIS 
functionality  and  user  requirements,  it  may  still  face  additional  costs  and 
risks  to  operational  readiness  and  availability.  If  DOD  develops  these 
performance  targets  and  metrics  as  part  of  a  performance-measurement 
process,  this  action  should  address  our  recommendation. 

DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to  develop  a  software  reliability 
and  maintainability  assessment  process,  with  metrics,  by  which  the 
program  can  monitor  and  determine  the  effect  that  software  issues  may 
have  on  overall  F-35  reliability  and  maintainability  (R+M)  issues.  DOD 
stated  that  the  F-35  program  has  an  established  process  for  assessing 
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software  reliability  and  will  continue  to  monitor  and  determine  the  effect 
that  software  issues  may  have  on  overall  F-35  R+M.  DOD  noted  that  this 
process  includes  a  Failure  Reporting  Analysis  Corrective  Action  System 
database,  a  Joint  Reliability  and  Maintainability  Evaluation  Team,  and  a 
Test  Data  Scoring  Board,  and  uses  the  Mean-Flight-Hours-Between- 
Failure  metric.  Although  we  agree  that  the  components  DOD  lists  are  in 
place,  and  that  DOD  does  have  a  process  for  assessing  F-35  R+M,  these 
actions  do  not  fully  address  the  intent  of  our  recommendation  because, 
as  we  state  in  our  report,  this  process  is  aimed  primarily  at  assessing  the 
reliability  of  the  F-35  aircraft’s  hardware  and  not,  specifically,  its 
associated  software.  Specifically,  the  process  does  not  determine  and 
monitor  software  reliability  using  software-specific  metrics  and  targets. 

We  continue  to  believe  that  DOD’s  current  process  and  associated 
metrics  do  not  allow  program  officials  to  have  a  full  understanding  of  the 
root  causes  of  reliability  issues  and  the  extent  to  which  software  failures 
contribute  to  these  issues.  As  our  report  states,  the  reliability  of  the 
aircraft  is  measured  with  a  number  of  different  metrics;  however, 
according  to  DOD  officials,  the  current  process  and  associated  metrics 
are  almost  exclusively  dedicated  to  hardware.  Since  the  F-35  is  the  most 
software-intensive  aircraft  DOD  has  procured  to  date,  establishing  a 
software  reliability  assessment  process  with  metrics  and  targets,  outside 
of  hardware  reliability,  would  allow  DOD  to  determine  the  extent  to  which 
software  failures  contribute  to  the  overall  R+M  of  the  F-35.  If  DOD 
develops  such  a  process,  this  action  should  address  our 
recommendation. 

DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to,  in  order  to  promote 
competition,  address  affordability  and,  to  inform  its  overarching 
sustainment  strategy,  develop  a  long-term  IP  Strategy  to  include,  but  not 
be  limited  to,  the  identification  of  (1)  current  levels  of  technical  data  rights 
ownership  by  the  federal  government  and  (2)  all  critical  technical  data 
needs  and  their  associated  costs.  DOD  stated  that  the  F-35  acquisition 
strategy  includes  a  data-management  strategy  whereby  it  will  continue  to 
assess  the  acquisition  of  technical  data  as  specific  needs  arise 
throughout  the  life  cycle  and  will  determine  the  level  of  specified  rights, 
critical  technical  data  needs  and  associated  costs,  through  business-case 
analysis,  contract  strategy  development,  and  ongoing  sustainment 
planning.  If  DOD  develops  a  long-term  IP  strategy  that  includes  the 
identification  of  its  current  level  of  specified  rights,  critical  technical  data 
needs,  and  associated  costs,  this  action  should  address  the 
recommendation. 
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DOD  concurred  with  the  recommendation  that  the  Secretary  of  Defense 
direct  the  F-35  Program  Executive  Officer  to  clearly  document 
assumptions  related  to  intermediate-level  maintenance  costs  and  revise 
assumptions  related  to  fuel  burn  rates,  part  replacement,  and  depot- 
maintenance  induction  in  its  future  F-35  O&S  cost  estimates.  DOD  stated 
that  the  Program  Executive  Officer  will  document  intermediate-level 
maintenance  cost  assumptions  and  revise  fuel  burn  rates,  part 
replacement,  and  depot-maintenance  induction  assumptions  in  future 
annual  updates  to  the  F-35  O&S  cost  estimate.  We  agree  that,  if  the 
assumptions  are  revised  and  fully  documented  to  reflect  the  current  and 
future  state  of  the  program,  this  action  should  address  the 
recommendation. 

DOD  concurred  with  the  recommendation  that  the  Program  Executive 
Officer  conduct  uncertainty  analysis  on  future  F-35  O&S  cost  estimates. 
DOD  stated  that  it  will  ensure  that  the  Program  Executive  Officer 
conducts  risk  and  sensitivity  analysis,  in  accordance  with  DOD  guidance, 
in  future  annual  updates  to  the  cost  estimate.  If  the  Program  Executive 
Officer  performs  an  uncertainty  or  quantitative  risk  analysis  on  future  JPO 
F-35  O&S  estimates,  this  action  should  address  the  recommendation. 

DOD  concurred  with  the  recommendation  that  CAPE  clearly  document 
assumptions  related  to  intermediate-level  maintenance  and  revise  its 
O&S  cost  estimate’s  fuel  burn  assumptions  to  better  reflect  the  current 
and  future  state  of  the  F-35  program.  DOD  stated  that  CAPE’s  current 
documentation  of  the  O&S  cost  estimate  reflects  an  approved  two-level 
maintenance  strategy  (i.e.,  organizational-  and  depot-level  maintenance 
only,  no  intermediate-level  maintenance).  DOD  also  stated  that  CAPE  will 
ensure  that  if  the  strategy  is  revised  to  include  any  planned  intermediate- 
level  maintenance  activities,  the  activities  and  associated  costs  will  be 
fully  captured  within  program  estimates  and  documentation.  However,  the 
program’s  maintenance  planning  documents  already  state  that 
intermediate-level  maintenance  will  be  used  in  the  sustainment  of  the  F- 
35,  and  it  is  currently  being  employed.  In  its  written  response,  DOD 
further  stated  that  any  intermediate-level  costs  are  already  included  in  the 
estimate’s  mission  personnel  costs.  While  DOD  guidance  states  that 
intermediate-level  costs  can  be  combined  with  unit-level  costs,  including 
mission  personnel,  the  guidance  also  states  that  this  should  be  noted  in 
the  cost  estimate’s  supporting  documentation  to  avoid  an  interpretation 
that  a  portion  of  the  maintenance  costs  were  omitted.  However,  as  we 
note  in  the  report,  CAPE  did  not  document  intermediate-level 
maintenance  costs  in  the  mission  personnel  or  unit-level  maintenance 
sections  of  the  cost  estimates.  Without  clear  documentation  of  these 
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intermediate-level  maintenance  costs,  it  is  unclear  whether  they  are  fully 
represented  in  the  CAPE  estimate.  If  CAPE  clearly  documents  where  and 
how  intermediate  maintenance  costs  are  included  in  its  future  cost 
estimates,  this  action  should  address  the  recommendation. 

Related  to  revising  its  O&S  cost  estimate’s  fuel  burn  assumptions,  DOD 
stated  that  CAPE  will  update  its  estimate  to  reflect  the  latest  approved 
technical  baseline  and  incorporate  the  latest  approved  fuel  consumption 
actuals,  as  this  information  becomes  available.  As  our  report  states,  the 
estimate’s  fuel  burn  rate  assumption  should  reflect  the  most  likely  future 
state  of  the  program  across  its  life  cycle.  To  date,  fuel  consumption 
actuals  are  from  aircraft  flown  using  limited  capabilities — at  slower  speeds 
and  at  lower  altitudes  than  the  F-35  will  eventually  fly — and  do  not  reflect 
the  most  likely  future  state  of  the  program.  We  agree  that,  once  fuel 
consumption  data  from  aircraft  flown  using  the  full  capability  are  available, 
these  data  could  be  used  to  update  the  assumptions.  Until  then,  the 
estimate’s  fuel  burn  rate  assumptions  should  be  revised  to  reflect  the 
future  state  of  the  program.  If  fully  implemented,  DOD’s  proposed  actions 
to  update  its  estimate  should  address  the  recommendation. 

Finally,  DOD  partially  concurred  with  the  recommendation  that  CAPE 
conduct  uncertainty  analyses  to  understand  the  total  potential  range  of 
costs  and  to  determine  if  its  estimate  reflects  the  most  likely  costs 
associated  with  the  program.  DOD  stated  that  the  department  agrees  with 
the  value  of  understanding  potential  cost  ranges,  but  stated  that  it 
considers  the  risk  and  sensitivity  analyses  regularly  performed  by  CAPE 
to  be  a  form  of  uncertainty  analysis.  As  our  report  states,  we 
acknowledge  that  CAPE  has  conducted  some  sensitivity  analyses  on 
previous  F-35  O&S  estimates,  but  it  has  not  conducted  an  uncertainty  or 
quantitative  risk  analysis  on  its  F-35  O&S  cost  estimate.  According  to 
DOD  guidance,  sensitivity  analyses  are  useful  for  identifying  critical 
estimating  assumptions,  but  they  have  limited  utility  in  providing  a 
comprehensive  sense  of  overall  uncertainty.  In  contrast,  an  uncertainty  or 
quantitative  risk  analysis,  such  as  a  Monte  Carlo  simulation,  can  provide 
a  broad  overall  assessment  of  variability  in  the  cost  estimate.  Similarly, 
the  GAO  Cost  Estimating  and  Assessment  Guide  states  that  risk  and 
uncertainty  analyses  should  be  performed  to  determine  the  level  of  risk 
associated  with  an  estimate.  We  continue  to  believe  that  without  an 
uncertainty  analysis,  DOD  cannot  fully  understand  how  much  of  the  total 
potential  O&S  costs  are  accounted  for  in  its  nearly  $1  trillion  estimate.  If 
CAPE  performs  an  uncertainty  analysis  on  future  estimates,  this  action 
should  address  our  recommendation. 
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We  are  sending  copies  of  this  report  to  appropriate  congressional 
committees;  the  Secretary  of  Defense;  the  Secretaries  of  the  Air  Force, 
Army,  and  Navy;  and  the  Commandant  of  the  Marine  Corps.  In  addition, 
the  report  is  available  at  no  charge  on  the  GAO  website  at 
http://www.gao.gov. 

If  you  or  your  staff  has  any  questions  about  this  report,  please  contact  me 
at  (202)  512-5431  or  russellc@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  Staff  members  making  key  contributions  to  this  report  are 
listed  in  appendix  IV. 


Cary  Russell,  Director 

Defense  Capabilities  and  Management 
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To  address  both  of  our  objectives — that  is,  to  evaluate  F-35  sustainment 
planning  and  risk  and  operating  and  support  (O&S)  cost  estimates — we 
collected  and  analyzed  information  and  interviewed  officials  from  the 
following  Department  of  Defense  (DOD)  offices: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisitions,  Technology 
and  Logistics); 

•  Office  of  the  Assistant  Secretary  of  Defense  (Logistics  and  Materiel 
Readiness); 

•  Office  of  the  Director  for  Cost  Assessment  and  Program  Evaluation 
(CAPE); 

•  Office  of  the  Director  for  Operational  Test  and  Evaluation  (DOT&E); 

•  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for 
Developmental  Test  and  Evaluation; 

•  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems 
Engineering; 

•  Department  of  the  Air  Force; 

•  Department  of  the  Navy; 

•  Headquarters  Marine  Corps;  and 

•  F-35  Joint  Program  Office  (JPO). 

We  conducted  visits  to 

•  Eglin  Air  Force  Base,  Florida; 

•  Marine  Corps  Air  Station  Yuma,  Arizona; 

•  Naval  Air  Station  Patuxent  River,  Maryland; 

•  Naval  Air  Systems  Command,  Maryland;  and 

•  Lockheed  Martin,  Fort  Worth,  Texas. 

To  assess  the  extent  to  which  DOD  has  developed  a  sustainment 
strategy  for  the  F-35  program  and  addressed  potential  risks  related  to 
affordability  and  operational  readiness,  we  reviewed  documentation  of 
program  plans  and  analyses  with  relevant  sustainment  elements, 
including  the  F-35  Life  Cycle  Sustainment  Plan,  the  Weapon  System 
Planning  Document,  the  F-35  Future  Support  Construct,  the  F-35 
Autonomic  Logistics  Global  Sustainment  Concept  of  Operations,  and  the 
F-35  Operational  Requirements  Document.  We  reviewed  documentation 
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of  ongoing  analyses,  including  Level  of  Repair  Analyses  and  Business 
Case  Analyses.  We  compared  these  documents  to  DOD  acquisition 
guidance  and  policy  on  sustainment  planning,  including  the  Defense 
Acquisition  Guidebook,  Interim  DOD  Instruction  5000.02  “Operation  of  the 
Defense  Acquisition  System,”  and  DOD’s  Life  Cycle  Sustainment  Plan 
and  guidance. 

To  determine  the  extent  to  which  DOD  has  developed  plans  to  mitigate 
sustainment  risks,  we  conducted  site  visits  (see  locations  above)  and 
interviewed  key  DOD  officials  in  order  to  collect  information  about 
ongoing  operations,  testing,  and  risks  to  the  sustainment  of  the  F-35. 

After  identifying  the  three  most  significant  risks  to  the  sustainment  of  the 
program,  we  compared  DOD’s  risk-mitigation  efforts  to  DOD  Risk 
Management  Guide  and  other  DOD  acquisition  guidance  and  DOD  policy 
and  guidance.  Specifically,  we  reviewed  DOD’s  efforts  to  monitor  and 
improve  the  risks  of  (1)  the  performance  of  the  Autonomic  Logistics 
Information  System  (ALIS),  (2)  the  reliability  and  maintainability  (R+M)  of 
the  aircraft,  and  (3)  the  management  of  technical  data.  We  then 
compared  these  efforts  to  DOD’s  Risk  Management  Guide,  Interim  DOD 
Instruction  5000.02,  and  other  specific  guidance  and  best  practices, 
including  DOD’s  System  of  Systems  Guide,  DOD’s  Manual  for  Improving 
Reliability,  Availability,  and  Maintainability,  and  Naval  Air  Systems 
Command  Software  Logistics  Primer. 

To  determine  the  extent  to  which  DOD  has  developed  a  reliable  operating 
and  support  (O&S)  cost  estimate  for  the  F-35  program,  we  evaluated  both 
DOD’s  Joint  Program  Office  (JPO)  and  its  Cost  Assessment  and  Program 
Evaluation  (CAPE)  office  2013  O&S  cost  estimates  using  GAO’s  Cost 
Estimating  and  Assessment  Guide.''  Specifically,  the  methodology 
outlined  in  the  Cost  Estimating  and  Assessment  Guide  is  a  compilation  of 
best  practices  that  federal  cost-estimating  organizations  and  industry  use 
to  develop  and  maintain  reliable  cost  estimates  throughout  the  life  cycle 
of  an  acquisition  program.  The  JPO’s  estimate  is  used  to  drive  budget 
planning  and  CAPE’s  estimate  is  used  to  assess  affordability  and  is 
required  at  major  milestones  or  by  special  request.  Generally,  either  the 
JPO’s  or  CAPE’S  estimate  is  used  as  DOD’s  estimate  of  record  and 
published  in  the  Selected  Acquisition  Report  (SAR) — a  report  that  DOD  is 


^GAO,  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Deveioping  and 
Managing  Capitai  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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generally  required  to  submit  annually  to  Congress  on  each  of  its  major 
defense  acquisition  programs.  For  the  F-35,  DOD  decided  to  use  the 
CAPE  O&S  estimate  in  the  2012  and  2013  SARs.  We  evaluated  both  the 
JPO  and  the  CAPE  O&S  estimates  because  both  estimates  are  used  in 
DOD  decision  making.  According  to  the  GAO  Cost  Estimating  and 
Assessment  Guide,  there  are  four  general  characteristics  of  sound  cost 
estimating:  being  well-documented,  comprehensive,  accurate,  and 
credible.  For  the  purposes  of  this  engagement,  we  conducted  a  limited 
assessment  and  evaluated  two  of  these  characteristics:  comprehensive 
and  credible.  We  chose  to  evaluate  the  estimates’  comprehensiveness 
because  if  a  cost  estimate  is  not  comprehensive  then  it  cannot  fully  meet 
the  characteristics  of  being  well-documented  or  accurate.  For  example,  if 
the  cost  estimate  is  missing  some  cost  elements,  then  the  documentation 
will  be  incomplete  and  the  estimate  will  be  inaccurate.  We  also  included 
an  assessment  of  the  credible  characteristic  due  to  the  substantial 
estimated  cost  of  the  program  and  the  need  to  better  understand  and 
quantify  the  F-35  O&S  cost  risk.  To  determine  whether  the 
comprehensive  and  credibility  characteristics  were  met,  we  reviewed 
CAPE  and  JPO  cost-estimating  documentation,  including  data  sources, 
assumptions,  and  calculations,  and  we  interviewed  cost-estimating 
officials  from  these  offices.  In  addition,  to  determine  how  much  more 
frequently  parts  were  being  replaced  than  assumed  in  the  JPO  cost 
estimate  (see  table  2),  we  compared  the  mean  flight  hour  between 
removals  (MFHBR)  data  for  the  Air  Force’s  F-35A  (CTOL)  and  the  Marine 
Corps’  F-35B  (STOVL)  from  a  12-month  period  ending  in  March  31, 2014, 
to  the  MFHBR  assumptions  used  in  the  JPO  estimate. 

We  also  gathered  F-35  reliability  and  maintainability  (R+M)  data^  from 
2013  through  2014  that  had  been  verified  through  DOD’s  Joint  Reliability 
and  Maintainability  Evaluation  Team  (JRMET).  To  determine  the  reliability 
of  these  data,  we  collected  information  on  how  the  data  were  collected, 
managed,  and  used  through  a  survey  and  interviews  with  relevant  DOD 
officials.  In  addition  to  the  data-reliability  survey,  we  also  reviewed  user 
manual  and  related  documentation  to  determine  the  limitations  of  the 
data.  By  assessing  this  information  against  GAO  data-quality  standards, 
we  determined  that  the  data  presented  in  our  findings  were  sufficiently 


^R+M  data  measure  aircraft  performance  to  determine  how  often  the  aircraft  experiences 
failures  and  how  much  time  it  takes  to  repair  those  failures. 
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reliable  for  presenting  information  about  the  aircraft’s  reliability  in  this 
report. 

We  conducted  this  performance  audit  from  August  201 3  through 
September  2014  in  accordance  with  generally  accepted  government 
auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 
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We  completed  an  overall  assessment  of  the  Joint  Program  Office  (JPO) 
and  Cost  Assessment  and  Program  Evaluation  (CAPE)  F-35  O&S 
estimates  on  the  basis  of  two  characteristics — comprehensiveness  and 
credibility — and  their  associated  best  practices  derived  from  the  GAO 
Cost  Estimating  and  Assessment  Guide.  After  reviewing  documentation 
that  JPO  and  CAPE  submitted  for  their  2013  F-35  O&S  cost  estimates, 
conducting  interviews  with  JPO  and  CAPE  cost-estimating  officials,  and 
reviewing  relevant  sources,  we  determined  that  these  cost  estimates  are 
not  fully  reliable.  While  the  estimates  are  comprehensive,  not  all  of  the 
assumptions  that  the  estimates  were  based  on  were  clearly  documented 
or  reasonable.  In  addition,  the  estimates  are  not  fully  reliable  in  that  the 
JPO  estimate  is  partially  credible  and  the  CAPE  estimate  is  minimally 
credible.  These  evaluations  are  shown  in  table  3  below.  We  determined 
the  overall  assessment  rating  by  assigning  each  individual  best  practice 
rating  a  number:  Not  Met  =  1 ,  Minimally  Met  =  2,  Partially  Met  =  3, 
Substantially  Met  =  4,  and  Met  =  5.  Then,  we  took  the  average  of  the 
individual  best  practice  assessment  ratings  to  determine  the  overall  rating 
for  each  of  the  two  characteristics.  The  resulting  average  becomes  the 
Overall  Assessment  as  follows:  Not  Met  =  1.0  to  1.4,  Minimally  Met  =  1.5 
to  2.4,  Partially  Met  =  2.5  to  3.4,  Substantially  Met  =  3.5  to  4.4,  and  Met  = 
4.5  to  5.0.  A  cost  estimate  is  considered  reliable  if  the  overall  assessment 
ratings  for  each  of  the  two  characteristics  are  substantially  or  fully  met.  If 
any  of  the  characteristics  are  not  met,  minimally  met,  or  partially  met,  then 
the  cost  estimate  does  not  fully  reflect  the  characteristics  of  a  high-quality 
estimate  and  cannot  be  considered  reliable. 
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Table  3:  Summary  Assessment  of  Joint  Program  Office  (JPO)  and  Cost  Assessment  and  Program  Evaluation  (CAPE)  Cost 
Estimates 

Characteristic 

Overall 

assessment 

Best  practice^ 

Individual  assessment 

Comprehensive 

Met  (JPO) 

Met  (CAPE) 

The  cost  estimate  includes  all  operating  and  support  costs. 

Met  (JPO) 

Met  (CAPE) 

The  cost  estimate  completely  defines  the  program,  reflects  the 
current  schedule,  and  is  technically  reasonable. 

Met  (JPO) 

Met  (CAPE) 

The  cost  estimate  work  breakdown  structure  (WBS)  is  product- 
oriented,  traceable  to  the  statement  of  work/objective,  and  at  an 
appropriate  level  of  detail  to  ensure  that  cost  elements  are  neither 
omitted  nor  double-counted. 

Met  (JPO) 

Met  (CAPE) 

The  estimate  is  based  on  reasonable  and  fully  documented  cost- 
influencing  ground  rules  and  assumptions. 

Partially  Met  (JPO) 
Partially  Met  (CAPE) 

Credible 

Partially  Met  (JPO) 
Minimally  Met 
(CAPE) 

The  cost  estimate  includes  a  sensitivity  analysis  that  identifies  a 
range  of  possible  costs  based  on  varying  major  assumptions, 
parameters,  and  data  inputs. 

Met  (JPO) 

Partially  Met  (CAPE) 

A  risk  and  uncertainty  analysis  was  conducted  that  quantified  the 
imperfectly  understood  risks  and  identified  the  effects  of  changing 
key  cost  driver  assumptions  and  factors. 

Not  met  (JPO) 

Not  met  (CAPE) 

Major  cost  elements  were  cross-checked  to  see  whether  results 
were  similar. 

Minimally  Met  (JPO) 
Partially  Met  (CAPE) 

An  independent  cost  estimate  was  conducted  by  a  group  outside 
the  acquiring  organization  to  determine  whether  other  estimating 
methods  produce  similar  results. 

Met  (JPO) 

Not  applicable  (CAPE)*’ 

Source:  GAO  analysis  of  JPO  and  CAPE  documentation  and  data.  |  GAO-14-778 


^Best  practices  are  derived  from  GAO,  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for 
Developing  and  Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

‘’We  excluded  the  best  practice  of  developing  an  independent  cost  estimate  from  our  evaluation  of 
CAPE’S  estimate.  It  would  be  unreasonable  to  expect  the  CAPE  estimate  to  have  its  own 
independent  cost  estimate  because  the  CAPE  estimate  was  developed  as  an  independent  cost 
estimate  of  the  JPO  estimate. 
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LOGISTICS  AND 
MATERIEL  READINESS 


ASSISTANT  SECRETARY  OF  DEFENSE 
3500  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3500 


SEP  a  ;^0i4 


Mr.  Cary  Russell, 

Director.  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Russell: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report,  GAO- 14- 
778,  “F-35  JOINT  STRIKE  FIGHTER:  Need  for  Affordable  Sustainment  Strategy  Greater 
Attention  to  Risks  and  Improved  Cost  Estimates,”  dated  August  1 1,  2014  (GAO  Code  351852). 
Detailed  comments  on  the  report  recommendations  are  enclosed. 


Sincerely, 

Paul  D.  Peters 
Principal  Deputy 


Enclosure: 
As  stated 
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GAO  Draft  Report  Dated  August  11, 2014 
GAO-14-778  (GAO  CODE  351852) 

“F-35  JOINT  STRIKE  FIGHTER:  NEED  FOR  AFFORDABLE  SUSTAINMENT 
STRATEGY  GREATER  ATTENTION  TO  RISKS  AND  IMPROVED  COST 

ESTIMATES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION  1:  To  help  the  Department  of  Defense  develop  an  affordable 
sustainment  strategy  for  the  F-35,  we  recommend  that  the  Secretary  of  Defense  direct  the  Under 
Secretary  of  Defense  for  Acquisitions,  Technology,  and  Logistics  to  direct  the  F-35  Program 
Executive  Officer  to  establish  affordability  constraints  linked  to,  and  informed  by,  military 
budgets  that  will  help  guide  sustainment  decisions,  prioritize  requirements,  and  identify 
additional  areas  for  savings  by  March  2015.  at  which  point  the  Future  Support  Construct 
decision  will  be  approved. 

DoD  RESPONSE:  Concur,  To  guide  development  of  an  affordable  strategy,  the  Department 
established  F-35  affordability  targets  for  unit  recurring  flyaway  cost  and  for  sustainment  cost  in 
March  2012.  The  F-35  program  is  currently  operating  under  these  affordability  targets  and  using 
affordability  goals  to  reduce  life  cycle  costs. 

RECOMMENDATION  2:  To  enable  the  Department  of  Defense  to  better  identify,  address, 
and  mitigate  performance  issues  with  Autonomic  Logistics  Information  System  (ALIS)  that 
could  have  an  impact  on  affordability,  as  well  as  readiness,  establish  a  performance  measurement 
process  for  ALIS  that  includes,  but  is  not  limited  to,  performance  metrics  and  targets  that  (1)  are 
based  on  intended  behavior  of  the  system  in  actual  operations  and  (2)  tie  system  performance  to 
user  requirements. 

DoD  RESPONSE:  Concur.  Through  the  remainder  of  system  development  and  demonstration 
(SDD),  the  F-35  program  will  continue  to  use  ALIS  performance  metrics  and  targets  to  assess 
impacts  to  the  operational  requirements,  such  as  the  sortie  generation  rate  key  performance 
parameter.  While  the  F-35  program  continues  to  assess  these  impacts  today,  future  incremental 
ALIS  functional  deliveries  and  additional  operational  experience  will  enable  more  refined 
metrics  and  targets  over  time.  These  refined  metrics  and  targets  will  help  reduce  performance 
risk  and  ensure  a  successful  operational  test  and  evaluation  at  the  end  of  SDD. 

RECOMMENDATION  3:  To  develop  a  high  level  of  confidence  that  the  aircraft  will  achieve 
its  reliability  and  maintainability  goals,  develop  a  software  reliability  and  maintainability 
assessment  process,  with  metrics,  by  which  the  program  can  monitor  and  determine  the  effect 
that  software  issues  may  have  on  overall  F-35  reliability  and  maintainability  issues. 

DoD  RESPONSE:  Concur.  The  F-35  program  has  an  established  process  for  assessing 
software  reliability  and  will  continue  to  monitor  and  determine  the  effect  that  software  issues 
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may  have  on  overall  F-35  reliability  and  maintainability.  This  process  includes  a  Failure 
Reporting  Analysis  Corrective  Action  System  database,  a  Joint  Reliability  and  Maintainability 
Evaluation  Team,  a  Test  Data  Scoring  Board,  and  uses  the  Mean-Flight-Hours-Between-Failure 
metric. 

RECOMMENDATION  4:  To  promote  competition,  address  affordability,  and  inform  its 
overarching  sustainment  strategy,  develop  a  long-term  IP  Strategy  to  include,  but  not  be  limited 
to,  the  identification  of: 

•  Current  levels  of  technical  data  rights  ownership  by  the  federal  government. 

•  All  critical  technical  data  needs  and  their  associated  costs. 

DoD  RESPONSE:  Concur.  The  F-35  acquisition  strategy  includes  a  data  management  strategy 
whereby  the  program  will  continue  to  assess  the  acquisition  of  technical  data  as  specific  needs 
arise  throughout  the  life  cycle.  Furthermore,  the  program  will  determine  the  level  of  specified 
rights,  critical  technical  data  needs  and  associated  costs,  through  business  case  analysis,  contract 
strategy  development,  and  ongoing  sustaiiunent  planning. 

RECOMMENDATION  5:  To  improve  the  reliability  of  the  JPO  F-35  O&S  cost  estimate, 
clearly  document  assumptions  related  to  intermediate-level  maintenance  costs  and  revise 
assumptions  related  to  fuel  bum  rates,  part  replacement,  and  depot-maintenance  induction  in  its 
future  F-35  O&S  cost  estimates  to  better  reflect  the  current  and  future  state  of  the  F-35  program. 

DoD  RESPONSE:  Concur.  The  F-35  Program  Executive  Officer  (PEO)  will  document 
assumptions  related  to  intermediate-level  maintenance  costs  in  future  annual  updates  to  the  JPO 
O&S  cost  estimate.  Current  documentation  reflects  an  approved  two-level  maintenance  strategy 
(i.e.,  organizational  and  depot  levels  only,  with  no  intermediate  level  maintenance).  The  F-35 
PEO  will  also  revise  assumptions  related  to  fuel  bum  rates,  part  replacement,  and  depot 
maintenance  induction  in  future  annual  updates  to  the  JPO  O&S  estimate. 

RECOMMENDATION  6:  To  understand  the  potential  range  of  costs  associated  with  the  JPO 
F-35  O&S  cost  estimate,  conduct  uncertainty  analysis  on  future  JPO  estimates. 

DoD  RESPONSE:  Concur.  The  Department  will  ensure  that  the  F-35  PEO  conducts  risk  and 
sensitivity  analysis,  according  to  DoD  guidance,  in  future  annual  updates  to  the  JPO  F-35  O&S 
cost  estimate. 

RECOMMENDATION  7:  Clearly  document  assumptions  related  to  intermediate-level 
maintenance  and  revise  fuel  bum  assumptions  to  better  reflect  the  current  and  future  state  of  the 
F-35  program. 

DoD  RESPONSE:  Concur.  The  current  documentation  of  the  CAPE  O&S  estimate  reflects  an 
approved  two-level  maintenance  strategy  (i.e.,  organizational  and  depot  levels  only,  with  no 
intermediate  level  maintenance).  While  the  program  does  not  pursue  an  1-level  maintenance 
strategy,  the  draft  report  correctly  notes  that  the  program  does  conduct  some  off-aircraft  (or 
“04-”)  maintenance  activities,  such  as  wheels,  tires,  batteries,  and  support/test  equipment,  as 
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defined  in  the  F-35  Life  Cycle  Sustainment  Plan,  Annex  A,  Program  Support  Strategy. 

However,  the  costs  for  these  personnel  are  already  captured  in  the  estimate  under  Mission 
Personnel;  these  personnel  are  part  of  the  Manpower  Estimate  Report  (MER)  provided  by  the 
Services  to  identify  their  squadron  manning  requirements.  CAPE  will  ensure  that  if  the 
maintenance  strategy  is  revised  to  include  any  planned  intermediate-level  maintenance  activities, 
those  activities  and  their  associated  costs  will  be  fully  captured  within  the  program's  O&S 
estimate  and  associated  documentation.  CAPE  will  also  update  its  estimate  to  reflect  the  latest 
approved  technical  baseline  and  incorporate  the  latest  approved  fuel  consumption  actuals, 
including  fuel  consumption  information  provided  from  the  flight  test  program,  as  this 
information  become  available. 

RECOMMENDATION  8:  Conduct  uncertainty  analysis  to  understand  the  potential  range  of 
costs  associated  with  its  estimates  to  reflect  the  most  likely  costs  associated  with  the  program. 

DoD  RESPONSE:  Partially  concur.  The  Department  considers  the  risk  and  sensitivity  analyses 
regularly  performed  by  CAPE  to  be  a  form  of  uncertainty  analysis,  as  described  in  the  2007  OSD 
CAIG  O&S  Cost  Estimating  Guide,  Section  5.3.4.  The  Department  agrees  with  the  value  of 
understanding  potential  O&S  cost  ranges  and  will  ensure  that  CAPE  continues  to  perform 
uncertainty  analyses  in  accordance  with  DoD  guidance. 
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